{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced Convolutional Neural Networks (CNN) - 1\n",
    "- Objective: try different structures of CNNs\n",
    "- Note: examples are performed on **i5 7600 + gtx 1060 6GB **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deeper Networks\n",
    "- It is one of definite trends that CNN models submitted to ILSVRC are getting deeper and deeper\n",
    "<img src=\"https://cdn-images-1.medium.com/max/756/1*Io-I-fOM0jGftDb_nne4KQ.png\" style=\"width: 600px\"/>\n",
    "\n",
    "<br>\n",
    "- Deeper networks are able to learn more \"abstract\" representations by comprising more operations and nonlinearities \n",
    "- However, they are difficult and costly to train and prone to overfitting\n",
    "\n",
    "<img src=\"http://cs231n.github.io/assets/cnnvis/filt1.jpeg\" style=\"width: 400px\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from keras.datasets import mnist\n",
    "from keras.utils.np_utils import to_categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Dataset\n",
    "- MNIST Dataset\n",
    "- Doc: https://keras.io/datasets/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# reshaping X data: (n, 28, 28) => (n, 28, 28, 1)\n",
    "X_train = X_train.reshape((X_train.shape[0], X_train.shape[1], X_train.shape[2], 1))\n",
    "X_test = X_test.reshape((X_test.shape[0], X_test.shape[1], X_test.shape[2], 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# converting y data into categorical (one-hot encoding)\n",
    "y_train = to_categorical(y_train)\n",
    "y_test = to_categorical(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28, 1)\n",
      "(10000, 28, 28, 1)\n",
      "(60000, 10)\n",
      "(10000, 10)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 0. Basic CNN model\n",
    "- Basic CNN similar to what we made before\n",
    "- One convolutional & pooling layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras import optimizers\n",
    "from keras.layers import Dense, Activation, Flatten, Conv2D, MaxPooling2D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def basic_cnn():\n",
    "    model = Sequential()\n",
    "    \n",
    "    model.add(Conv2D(input_shape = (X_train.shape[1], X_train.shape[2], X_train.shape[3]), filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same'))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size = (2,2)))\n",
    "\n",
    "    # prior layer should be flattend to be connected to dense layers\n",
    "    model.add(Flatten())\n",
    "    # dense layer with 50 neurons\n",
    "    model.add(Dense(50, activation = 'relu'))\n",
    "    # final layer with 10 neurons to classify the instances\n",
    "    model.add(Dense(10, activation = 'softmax'))\n",
    "    \n",
    "    adam = optimizers.Adam(lr = 0.001)\n",
    "    model.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['accuracy'])\n",
    "    \n",
    "    return model    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = basic_cnn()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_13 (Conv2D)           (None, 28, 28, 50)        500       \n",
      "_________________________________________________________________\n",
      "activation_4124 (Activation) (None, 28, 28, 50)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_8 (MaxPooling2 (None, 14, 14, 50)        0         \n",
      "_________________________________________________________________\n",
      "flatten_6 (Flatten)          (None, 9800)              0         \n",
      "_________________________________________________________________\n",
      "dense_11 (Dense)             (None, 50)                490050    \n",
      "_________________________________________________________________\n",
      "dense_12 (Dense)             (None, 10)                510       \n",
      "=================================================================\n",
      "Total params: 491,060\n",
      "Trainable params: 491,060\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 13min 30s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "history = model.fit(X_train, y_train, batch_size = 50, validation_split = 0.2, epochs = 100, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW9//HXJzOTfQXCkgQElC1sghGxVKvFKqh1ay1g\n1WpVbq1W29veXmvvo3az9dfberW3Xr3aurVVanGjXhTrVmvrAsgiO8iWhC0sIXsyy+f3x3cSQpjJ\nDJAInHyejwcPMmeZ+X5neZ/v+Z7vOUdUFWOMMT1HyrEugDHGmE+WBb8xxvQwFvzGGNPDWPAbY0wP\nY8FvjDE9jAW/Mcb0MBb8xhjTw1jwG2NMD2PBb4wxPYz/WBcglj59+ujgwYOPdTGMMeaEsXjx4t2q\nWpjMssdl8A8ePJhFixYd62IYY8wJQ0S2JLusdfUYY0wPY8FvjDE9jAW/Mcb0MMdlH38swWCQiooK\nmpqajnVRPCE9PZ2SkhICgcCxLoox5hOWMPhF5FHgYmCXqo6JMV+A+4ELgQbgOlX9MDpvWnSeD/it\nqt5zpAWtqKggJyeHwYMH417SHClVZc+ePVRUVDBkyJBjXRxjzCcsma6ex4FpncyfDgyL/psNPAgg\nIj7ggej8UmCWiJQeaUGbmpro3bu3hX4XEBF69+5te0/G9FAJg19V3wb2drLIpcCT6rwH5IvIAGAS\nsEFVN6pqCzAnuuwRs9DvOvZeGtNzdUUffzFQ3u5xRXRarOlnxHsSEZmN22Ng0KBBXVAsY7wjFI6w\nvHI/+xuCnD28EF9K1224g+EIAvh93TPWY199C+X7GshK85OXESA3PUCqv/vHlTSHwryxehfb9zcx\nriSPMcV5pAd83fZ64YiyaXcdhdnp5GUe38fOjpuDu6r6MPAwQFlZ2XF3I+Dq6mqeeuopvv71rx/W\nehdeeCFPPfUU+fn5cZf5wQ9+wNlnn8155513tMXsEht21fLjl1YzaXABXz/nFFLahYyqElHiBs+q\nbTU8+o9NrN9Vx5iiXE4dmM+4knwG98kkzX/wj05Vqaxu5MOt1WzdU88Fo/szrF9OzOfdvLuef3y8\nm+ZghFR/Cmn+FEb2z2V0UW5b+VpCET6qrKZiXyP7G4PUNAbJTvMzujiP0gG5ZKb6qG8Js7u2mR01\nTVTua6SyupHapiBZaX6y0/xkpvpp3RnypQiFOWn0y0knPzPAvoYWqmqb2VvfQjAcIRhWQuEI9S1h\napqCNLaEKR2Qy9nDCynKz2ir476GIHvq3HrVjUEiESXVn0KqP4VQRKlrClHXHKK+OURTMExTMEJL\nOEI4okRUqdzXyLsb91DbFAJgWN9s/vVzw5k2pj+1zSFWVO7n4111tIQVVUVEGNonixH9cxiQl075\n3kaWlO9j9fZaslJ99MtNp1dWKqu21/DPj3fz4ZZqWsIRMgI+stP99M1JY1CvTAb2yqRXVirp/hQy\nUt1n19gSpjEYwZcCuekBcjMCqEJldQPbqpvYU9/S9pnVNgVZs72WHTWHdimm+VPISXcbgvED85k8\ntDennVRAUzDM9uomdtQ0UdsUoqElRENLmFR/CtlpfnLS/eSmB9wGJMNPwJdCRF3oBsMRGoNhmlrC\nvL9pLy8sraS6Idj2mv4UoaQgg1R/Cv6UFPIyApw+uIDJQ3tzUp8slpVX88GmvazaVkNzKEww7N7/\n1u9GXkaA0UW5lA0uYExxHjv3N7O8sprlFftZWl7Nisr9NLSESREYPzCfs4cVMn1sf0b2zz2o7ut2\n1rJ5d33bY1+KkB7wkR7wkZXmO2T57iDJ3GxdRAYDL8U5uPu/wFuq+nT08VrgHGAw8ENVvSA6/XsA\nqvrzRK9XVlamHc/cXb16NaNGjUpY1u6yefNmLr74YlasWHHQ9FAohN9/3Gw/ARc29S1h6ptD+FPE\nhYwvhYA/hZR2XTyrVq0iv2gIeRkBstL8hCPK797ZyC9fXYcAzaEIF47tzy+vHE9GwMe8Zdv4xStr\n2VnTRFF+BgN7ZdAvJ53sdPeDXFpezT827CEz1ceYojxWb6+httmFlS9FGNQrkwF56TSHIjS0hNld\n10xVbfNBZT9rWB++fIbb46vY18jmPfW8s343m/c0xKxrQWaAT53ch5qmIIs276MxGI65nAik+30x\n52cEYk8/HK3vcV20vkMLsxBgW3XTYT+3L0UI+ASfCCkpQkFmKp86uTefHtYHVbjvtXV8XFVP76zU\ng4I2Zrl8KbSEI4ALvlDkwO9dBEoH5HLm0N7kpAeoaw5S2xRiR00T5XsbKN/XSEsoknS5c9L89M5O\nbfuOpQV8jOyfQ+mAXE7qnUljMNy2Qa5tDlHbFKKqtpnFW/axN049UsR9Pi3RDW2yUn0pfG50P648\nrYRRA3Kj4byPrXsbCUWfa1dtEysq99PuLSE9kMKYojyy0vwEfAIIjcEQdU0hdte1UFnd2PbetUZn\nqi+F0mgjZ0xxHlv31PO39btZXlGNqtsIzCgbSFMwzLMfVrByW03ccvfJTmXRf3wu6Xq2JyKLVbUs\nqWW7IPgvAm7Fjeo5A/i1qk4SET+wDpgKVAILgatUdWWi1zseg3/mzJm8+OKLjBgxgkAgQHp6OgUF\nBaxZs4Z169Zx2WWXUV5eTlNTE7fffjuzZ88GDlx+oq6ujunTp/PpT3+af/7znxQVFfHc8y+Qk53F\nddddx8UXX8z0z1/GiGEnc+kXZ/HmX18mFApx/8NPMnbMKJpq9jH7q1+hcts2xk88nXf+9gZ/fuVt\nevXujS9FSPOnkOb3IQLVDS00x/jBChDwpRDwuZZm+ab13PjidgDyMwNkpfqprG7k/NJ+3H35WF5Y\nUsnPX17NyP65pAVSWLK1mtFFrkVbsa+RrXsb2FPXTG20xdo3J42vfGows04fRF5mgEhE+biqjpXb\navi4qo6Pq+rYsb+JjFQfmamuBTWuJI8JAwvol5vGM4vKefLdLexqtzHISvUxaUgvzhnRl88MLyQ/\nM0BLyLWyl5VX8/f1u3lv4x5y0v1MHtqbyUN7c0rf7LYWYXVDkBWV+/mocj91TSEKc9Lok51Gv9x0\nigsyGJCXTnrARzii1DWHaGw5ENLBcIRdtc3sqmliX0OQXlkB+mSn0Ts7jTR/Cv4Uwe9LISvNR5rf\nh6qyflcdb6+r4t2P95DqT6E4P4Oi/AwKc9IoyEwlPzOAL0VoCUVoDkXw+4ScND856QEy03xkBHwE\nEnS5hCPK80sq+fv6Kob1zWZsST4j++eQ7vchKRAMRdiwq461O2vZsqeBIX2yOHVgPiP65xCOKFW1\nzVTVNTOkdxYFWalxXycSUZpD0VZ0dOOVHnBlDKtS0xhkf6NrURflZ5CXcWTdG63v29LyanLT/fTP\nc59LXkaANH8KIoKqK0ttU4iapmDba4cjSooIIi6A01Nd+ZItT21TkEVb9rFldz3jBuYzpiiv026o\n1g3Visr9FOVnMK4kj+H9cmKus6eumReWbuNPC7eybmcdAGOKc/nCxBJOH9yrbblwRGkKhtsaCOeM\n6Hu4byHQxcEvIk/jWvB9gJ3AXUAAQFUfig7n/A1u5E8DcL2qLoqueyFwH24456OqencyhUoU/D/6\ny0pWdbLVPBKlRbnc9fnRcee3b/G/9dZbXHTRRaxYsaJtOOTevXspKChg9/46zvrUZF5e8BonFfdn\n6NAhbcF/yimn8Obf36X4lFHc/NVr+MznpnHljKv4j299nbOmXsDZF1zM9DPHM/uW27jhX27myd89\nzLKlS/nB/7ufn3z/O/TrP4Abb/02C995gxtmXcGy9Vso6NWHcPQH2hKKoLhd016ZqeRmuPBtCbt5\nzdH/g9HA2b7lYza05FHTFKRyXyO7apu5cGx/Lju1uO3g75trd3HbU0tIT/XxbxeM4AsTS2J287R+\nj472oHFLKMKiLXvJTQ9QUuB+vHYg2pzIVJUVlTWkBVIYHqcrsyscTvAn7KNQ1VkJ5itwS5x584H5\nyRTkRBBRKN/bQHMwzKRJk9pCPxSO8LP/vJeX5r2IqrKtopx/LFlBgy+TcESp3NfAvv11FA88ibyB\nw2gJRTi97DRqqraTIkJTKEw4EqEoP4OAT7jxmpkUF2Qy9awzeeOVlygtymXVkg/4/p/+TGlRLmNn\nXs53vl5AUX4mfXpltiufEonoQQfpfClCwJ9CVtqh9WnYFeDqiSd1WudzR/Tl7e+e61p6qfEPjHVV\nOKf6U/jUyX265LmMOR6ICGNL8o51MQ5yfHVOJ6mzlnl3qWsOEgxH2Nfg+vl8qem0hCLsbWjh5QWv\n8frrr/Hs/NcpKsznsgvPp1e6kJseIKJKfXOYFBEy0tMZ3DuLnHQ/+Vnp1NXVcXLfbPIzUinKz6BP\ntkvntDT3v8/nIxQKISKkiJCdFuh0NEeKCCm+rm8dd9YdYIw58di1epKwv6GFfUE/DXV1DO+XQ05G\ngGA4wpodNeyqaSLUVE9RYR9Gn9SXnVs38v5775GZ6mdgr0wCvhSG989hUO8s/D4hN0bXhUjiFvOU\nKVN45plnAHj11VfZt29ft9XXGONtJ2SL/5Oi6g6E7axpoqhvIWefNYWyCePJyMigT2FfCnPSyM8I\nMOJLl/HcU48zatQoRowYweTJk7u8LHfddRezZs3i97//PWeeeSb9+/cnJ6f7+guNMd6V1KieT9qx\nGNXT2BKirjlMZqrryw6GIpTva6ShJUReRoCBBZkHjWf/pDU3N+Pz+fD7/bz77rvcfPPNLF269Kie\n81iPlDLGdJ0uPbjbE+xvDFK+t4FIdCPYOg5ZBAb1yjwuRpZs3bqVL33pS0QiEVJTU3nkkUeOaXmM\nMSeuHh/8e+qa2VbdSEaqn5KCDJqDYeqaw6gq/XLTCXwCp5YnY9iwYSxZsuRYF8MY4wE9NvhVlR01\nTVTVNpObHmBQL9eVkx7wkZeZeH1jjDlR9cjgD4YjlO9toK45RO/sNIry0o95V44xxnxSelzwNzSH\n2LK3gXBEGViQaWPUjTE9To8K/uZgmE176vGlCCcXZpGR2qOqb4wxQA86gSscibB5TwMCDO3T/aGf\nnZ0NwLZt2/jiF78Yc5lzzjmHjsNWO7rvvvtoaDhwZcoLL7yQ6urqriuoMabH6RHBr6qU73WXmB3U\nK4tUf/fdjKGjoqIi5s6de8Trdwz++fPnd3ptf2OMSaRHBP/O2mZqmoIU5btrxx+JO+64gwceeKDt\n8Q9/+EN++tOfMnXqVCZOnMjYsWN58cUXD1lv8+bNjBnjrmbd2NjIzJkzGTVqFJdffjmNjY1ty918\n882UlZUxevRo7rrrLgB+/etfs23bNs4991zOPfdcwF3meffu3QDce++9jBkzhjFjxnDfffe1vd6o\nUaO46aabGD16NOeff/5Br2OMMSdmJ/fLd8COj5JaVFFyWsLkR69Z765KH0P/sTD9nrjPM2PGDL75\nzW9yyy3uQqTPPPMMCxYs4LbbbiM3N5fdu3czefJkLrnkkrgjhB588EEyMzNZvXo1y5cvZ+LEiW3z\n7r77bnr16kU4HGbq1KksX76c2267jXvvvZc333yTPn0OvmLl4sWLeeyxx3j//fdRVc444ww+85nP\nUFBQwPr163n66ad55JFH+NKXvsSzzz7L1VdfndT7ZYzxPs+3+CPq7pTjTxEkXugnYcKECezatYtt\n27axbNkyCgoK6N+/P3feeSfjxo3jvPPOo7Kykp07d8Z9jrfffrstgMeNG8e4cePa5j3zzDNMnDiR\nCRMmsHLlSlatWtVped555x0uv/xysrKyyM7O5oorruDvf/87AEOGDOHUU08F4LTTTmPz5s1HXG9j\njPecmC3+TlrmHe2pcffvHDUgF47yZtJXXnklc+fOZceOHcyYMYM//vGPVFVVsXjxYgKBAIMHD6ap\n6dD7iyayadMmfvnLX7Jw4UIKCgq47rrrjuh5WrVe1hncpZ2tq8cY057nW/x1zSHSk7idXTJmzJjB\nnDlzmDt3LldeeSX79++nb9++BAIB3nzzTbZs2dLp+meffTZPPfUUACtWrGD58uUA1NTUkJWVRV5e\nHjt37uTll19uWycnJ4fa2tpDnuuss87ihRdeoKGhgfr6ep5//nnOOuuso66j6YGWzYHd6491Kcwn\nyNPBH4m4m47npHXNjs3o0aOpra2luLiYAQMG8OUvf5lFixYxduxYnnzySUaOHNnp+jfffDN1dXWM\nGjWKH/zgB5x22mkAjB8/ngkTJjBy5EiuuuoqpkyZ0rbO7NmzmTZtWtvB3VYTJ07kuuuuY9KkSZxx\nxhnceOONTJgwoUvqadqJhGHp0zDny7B9+cHzdq6EP1/n/u9O4VD3PfeWd+H5f4E/XAFN+7vvdU5U\nkXDiZU5Ayd5sfRpwP+7eub9V1Xs6zC8AHgVOBpqAr6rqiui8bwE3Agp8hLsnb6f9GF11WebapiCb\ndtczpE8WOelHdiNoLzthL8vcXAdb/gkDxkFO//jL7VoN1VuhuAyyeif//KFmaNgLW9+Ft34Ou9dB\nSgAkBab9DMpugA+fgJf/HUJNUDAYZv8NMrp4mK0qvPB12PQ3uHYe9DnlwLyNf4OPnoHzfwoZBUf+\n/I9fDDtXQHMtlF4KX3zUXZbWwMrn4cVvwGUPuPfmONell2UWER/wAPA5oAJYKCLzVLX90cc7gaWq\nermIjIwuP1VEioHbgFJVbRSRZ4CZwOOHVaMjVNfsbluYeaKeoRsJuR+nr5s3Ws11sH4BrHsVTr0K\nhn6m3bxamHMVRCIwfqb7AaTnunLV74ZdK6F8IVR8AP50OOvbUHRq7NfZuwkWPwbp+ZBX4gKzaCL4\nkvx81vwfLHsa1v/VBW56Hlx8H4y54tBl63fD4xdBwx73uNfJMPZK+My/Q0qcHd33H4Y3fgLNNQem\n9RkBX3oSBn0KXrgZ/u/b8N6DsGcDDD0XJt0Ez1zr5s34Y/znPhJ/+wUse8q9r09eAtfPd+/Zqhfh\n2Rsh3OI2bte84D6TzjTth21LoOR0SM1y0za+CVvegem/cJ/zGz+Bkz8LE685svJWfgirXoD8QdD7\nFFfWzN6Qmg0agV2roGIh1Gx371t234PXV0280Wmug0AGpBzluTj7tsC2DyG7P+QVQ86Ag39nO1e6\njW6oCZ77F8gbCMUHRuGxa7WrY3f/NrtJMr+4ScAGVd0IICJzgEuB9sFfCtwDoKprRGSwiPRr9xoZ\nIhIEMoFtXVX4ROqaQmSm+jq9T21CTTVQux3yB0KgGy7bGQ5BTSX40yC734EvfrjF9btqxIWPP8Y1\nhVQhHHQbCH+a+zGoQrARmqrdlzZngPuhxKIRFyCrX4JQI4jPhcrVc2HwpyHUAn+6Gjb/AwpOgnm3\nwvx/g5x+7scbbj7wXIUjoXYHrJ7nNg7n/gcUDj8wv7kW/nilC0za7WVmFcKoz7t1Bp4Ru6yRCLz6\nfXjvf9x7NPFaGHoO/P1emHs9rFsAF/7CbQhazf+Oe80vPupa/Zvehr/d497rz99/aHAs+QO8/G8w\n5DMw5CwXWHkDXRC2LnvVM/Duf8Pbv4KpP4Ap33JBf/5P4ZU74J/3w2nXwYbX3esNnATjZh7YsNXu\ngHcfgJKyxC3Ij+bCWz+D8VfBmV93LfMnPu/2Nl7/kQvw0653n8lTM9xn1hrorZr2w9qXYeUL8PHr\n7js14FT48p/d+/7GT10dT7sOUvxuz+Ll70K/0QeHXHvhkNsLqtsBn/sJZPZy0zf+DZ6eCcGGQ9dJ\nCbjnD7UbZLD4cfjCI+5zbKqBd/4LFv4OPn07fPpfY28Adq+H353vXvNzP4YRFx5YrqXB7ZEF0uO/\np5EwrH8VFj3qGg/tv4epOfCZ78IZX4NgvevaS8uBry5wfz89C256w33nX/kerHsF+o5236WBp8d/\nzcOx8S33Wzv3zm7f60rY1SMiXwSmqeqN0cfXAGeo6q3tlvkZkKGq3xKRScA/o8ssFpHbgbuBRuBV\nVf1yokLF6+oZOXJk0lfRDIUjrNpeQ//cdPrmdvJlSGT3emipc1+qgsEHh8vRaqpxoRQJuseZfVxL\nOBKGPevdDxUBXyr0GXYg2Bt2uxZtqJmDvrwpflfOcIt7LNHA6jUU0rIPemmNhFmz+B1Gzb8cyq6H\n0ZdD72EuXGoq4ernYOEj8NGf4dL/cXsCFYtg+Z+gcS/kFruy9j7ZdaVk5LugefcB9y8Sgssfcs+r\n6vrCV89zXRZFE6Bmm+tiWD3PBXewwZW//1i3ATh5qgtgBF74mtvtPuNmuODuA0EcDsHb/+n+9RoC\ns+a492nVPHjmGvjsf8DZ/xatsMKbP4O3f+Fa/pc9dCCQV//FtdqHnuOew59Gpzq2TFXdBmjVi668\nGgZ/hgu63sNcGXaugA8ecdNS/C58T/7sgefYtgTKP3B/BxtdWYtPg2tfcOWp/BCevNTtjZxyntsL\nSc2CFc+6jffAM2D4BZARDeK1Lx8I+9wSt6HpfTIs+L7bcE+aDQvuhEv+221IwW3M//csqK9yG5bx\nM6H0MsiKnkNSu9PVc8s/3Pcsuz9c/uCBBkLvU+Ca513992xwrerGva7bLByEAeNdSAYb4c/Xuy60\n8bNcGDfshr6lbq/g9BvdXkj7jXNdFfx2KrTUu+DfvQ5OmgJ9R7n3bedKV6b+Y90Gt98Y15LPLXEb\nqZUvuO9afZUr92lfgeHTXPn2V8La+S7M+wx3G8Xy9+G6/4NBk2HnKrfByShwz+VLhbKvuve+Zhuc\nfgOc90O3oYgn1Ox+6zWVbp0+I6DktAPzKxbBE5e4BtYNfz3k95qMw+nq6argz8UdA5iA68cfCdwE\nbAGeBWYA1cCfgbmq+ocYrzMbmA0waNCg0zqOkNm0aRM5OTn07t07qfCvbmhh694GTinMJvNID+6G\nmt0XMbOPawUEG13gZRUmv0UONrmASWlXhnDQtf4adrvd+PyToGkf1O1y3SDhFvdavU92rfK9G6Pd\nIwNh/1YXsIEs98NvbemHWiDc5DYaablueQ3Dno/d8xUMbuuD1kiEPVvXULvmLYb0y4UJ7U7uqt0B\nj013P1oNu5btWd8+vPetbhf86Roof8+tH8h0LeKpd8FZ/3ro8i0NrrVZ/r77AVQudhsCf4brw9+3\nybWqz7w19vu++R8uuMNBuPhe1yLL6e9aaB13xf/+K3j9x9BvrNuL86e5LqQB4+HaFw9tNSeruRb+\n7ztuYzh8mmsxr30Z3rzbfYcQGDcDzrzFdQvt2wJffcWF3T/+C964273frfoMh+tfOfjYROVi2PAG\nTLn94D3AZXNcF1RL3YFpucUutEdf5jbMrV1QFYvgqS+5LrBeQ+GWhQd3tdXtcs+37OlouXEhetIU\nt2Fr2g+X/NptYJ+9yTVQUvxuL+GaFw7sASTSUg/zvwtL/+Ce+/yfur2R1+6Cf/4aRl4M037uvvPB\nRnjiYhfA10U/qw8fhzd/7vZqiydCySTX2KhY6DaS7fcuwH0Hh18AY77gPp9YXTTrFrjjNvs2wUW/\nchugtnmvuo1b6aVubyN3QLR77G54/yFXpqufPbCRBFfetfPdd3vr+wfvISPwqW+4xsneje43l57n\n9jA6O27Via4O/jOBH6rqBdHH3wNQ1Z/HWV6ATcA44ALcRuOG6Lxrgcmq+vXOXjNWiz8YDFJRUZH0\n+PZ9DS00toQZkJdx5HtNTfvdv9wi15po2BttmQbcFjk1y02PRdV1tzTXAuK6MFIz3Re1ud4tk5bt\nArq1gE01bh3EfYFauz1ap0uKe96M/M5bF+1FIq6VE252LRV/GmiE9KrllGQGCZx9+6HrVJe7cDjl\nPPclP5I3MNjkuiE++rN7PHw6zHwquT7wYJPre163wP2IJ98MY2Nf6O5Ambe6YxE7PnJBdNOb7uBv\nLIsec8EWbHChkj8IvvC75EPrcETCri89byAUjnDT9le61iviusM2vuUC6fyfuoYAuI13ssc+wH0v\nWupdCzbU7I5pxHuv93wMf7kdpnwThp0X//l2LHddIq3BlVfi9jT6u0uQ0NIAr/3QvfeXP3RkB7f3\nV7iNVPvv2HsPuo036lrnGflQtRZm/AFGXXxgudYRNx277cJB17LeX+n+D2S4PcjUJLpqg03uuFXR\nxEO/9+Fg/A3GM9e6z/jaF1yD5fUfwYdPujr0GwtDznYbh7xi11353v+4Lqd+Y10DUNU1BHoNSeZd\ni6mrg98PrAOmApXAQuAqVV3Zbpl8oEFVW0TkJuAsVb1WRM7AjfY5HdfV8ziwSFX/u7PXjBX8h+uz\nv3yLoYXZ/PYrSb0Ph1KFX09wX/brXnLTIhFY9bzryqhc7EK77Hq325xbdGDdmu2ua6P8PddqEJ8L\nwMa9bqMx4cvuRxfrQ179kgvnYZ87uCzzvgFb34Mr/td1ARyOYCN88DB8/KZ7jlAjnH4TXPif3duX\nqApv/9IFx4zfH/nok2S11MNff+Ba0aff0L2vdbS2L3etvEjIdWtMvPb4Hk0TanGh90mVcdca2Px3\n142za5U7GHzadZ/Max+JLf90x1pSs9zvraXOHS+Y8k3ILoy9ztpX4MVb3Hfg+pehX+lRFaFLgz/6\nhBcC9+GGcz6qqneLyNcAVPWh6F7BE7gO55XADaq6L7ruj3BdPSFgCXCjqjbHeJk2XRH8p9/9GueN\n6svPr4jT6ktky7vw2DTXF3zqrIPnqbov5Lv/7boJJMUdaErxuVZG1RrXGrnk1wdaqqEWNzyw9ylu\nq38kkhn1kEio2fWP9h3dtSNQzOGrWuda9b2GHuuSmK6wfbnbU+4z3G3M+3Z+Xg8AjdXuN5nTL/Gy\nCXR58H/SuiL4J/z4VS4aN4CfXjY2/kKqrg891tCwF2+FFc/Bd9Z1fqBl7yZ4/39h5XNu2Fpeses6\nOPNWd+DJGNNzRCLHrEHVpeP4T1ShiOLv7APYuwmevcEdzLrswegIkqiWejcKYPRliY+u9xrirh10\nGNcPMsZ41AmyF+3d4A8rAV+0W+S5f4F9m93wtNGXw8dvuINbIm742xOfhym3ufHDlYvdGOqWWjeE\n0RhjPMa7wR+J4EtJcQddls9xQzBf+qY7QSXc4oa4ffFRN3pmwffhH/e7f+BGhYy+wp2taYwxHuPh\n4I+2+N9p/dRiAAAT/ElEQVS6B7L6wu3L3EHXj/7shu1N+eaBoVmfvw9GXuRGvAw60520cQQnUBhj\nzInAk8EfjiiqMKh2mRtKeP7dbgxv8cT4p6IP+9zBQyiNMcajTowjEYcpGI4AMLn8EdfFU/bVY1wi\nY4w5fngy+MMRpUzWMLD6A9elk8wZe8YY00N4MvgjlUv5YeBJGlN7WWvfGGM68Fbwb30f/vBFcp6c\nyiDZxfsjv2etfWOM6cA7B3ebauD3l0MgndpP38mU14ZyR/GkY10qY4w57ngn+NNz3TXOi05lf71Q\n+9qbBE6Qs+iMMeaT5K1kHDwFUrMIhd31h/y+4/hqh8YYc4x4K/ijQhEX/Ed1y0VjjPEojwa/G8cf\n8HmyesYYc1Q8mYxtXT3W4jfGmEN4M/gj1sdvjDHxeDP4o5ds6PR6/MYY00N5MhmDNqrHGGPiSir4\nRWSaiKwVkQ0ickeM+QUi8ryILBeRD0RkTLt5+SIyV0TWiMjq6P15u1W4tavHWvzGGHOIhMkoIj7g\nAWA6UArMEpGOt4O/E1iqquOAa4H72827H3hFVUcC44HVXVHwzgSjo3qsxW+MMYdKpkk8CdigqhtV\ntQWYA1zaYZlS4A0AVV0DDBaRfiKSB5wN/C46r0VVq7us9HG0juqxM3eNMeZQySRjMVDe7nFFdFp7\ny4ArAERkEnASUAIMAaqAx0RkiYj8VkSyjrrUCYSjLX47gcsYYw7VVU3ie4B8EVkKfANYAoRx1wKa\nCDyoqhOAeuCQYwQAIjJbRBaJyKKqqqqjKkzrwd2AdfUYY8whkgn+SmBgu8cl0WltVLVGVa9X1VNx\nffyFwEbc3kGFqr4fXXQubkNwCFV9WFXLVLWssLDwMKtxsFBbH7919RhjTEfJJONCYJiIDBGRVGAm\nMK/9AtGRO6nRhzcCb0c3BjuAchEZEZ03FVjVRWWPy87cNcaY+BJelllVQyJyK7AA8AGPqupKEfla\ndP5DwCjgCRFRYCVwQ7un+Abwx+iGYSNwfRfX4RB25q4xxsSX1PX4VXU+ML/DtIfa/f0uMDzOukuB\nsqMo42GzM3eNMSY+TyZjW4vfunqMMeYQ3gx+u2SDMcbE5cngD9r1+I0xJi5PJmNri99O4DLGmEN5\nM/itj98YY+LyZvCHI/hTBBELfmOM6cibwR9R6+Yxxpg4vBn8YbUDu8YYE4cn0zEUidhQTmOMicOT\nwR8Mqx3YNcaYODwZ/OFIxC7XYIwxcXgyHUNhta4eY4yJw5PBH4xYV48xxsTjyeAPRyJ2ExZjjInD\nk+loB3eNMSY+TwZ/KGzDOY0xJh5vBn9EbVSPMcbE4cl0dGfuWovfGGNi8Wbw2zh+Y4yJK6l0FJFp\nIrJWRDaIyB0x5heIyPMislxEPhCRMR3m+0RkiYi81FUF70woYuP4jTEmnoTBLyI+4AFgOlAKzBKR\n0g6L3QksVdVxwLXA/R3m3w6sPvriJidko3qMMSauZFr8k4ANqrpRVVuAOcClHZYpBd4AUNU1wGAR\n6QcgIiXARcBvu6zUCQTDNo7fGGPiSSYdi4Hydo8rotPaWwZcASAik4CTgJLovPuA7wKRzl5ERGaL\nyCIRWVRVVZVEseIL25m7xhgTV1c1i+8B8kVkKfANYAkQFpGLgV2qujjRE6jqw6papqplhYWFR1UY\n18dvLX5jjInFn8QylcDAdo9LotPaqGoNcD2AuPsdbgI2AjOAS0TkQiAdyBWRP6jq1V1Q9riC4QgB\na/EbY0xMyTSLFwLDRGSIiKQCM4F57RcQkfzoPIAbgbdVtUZVv6eqJao6OLreG90d+uC6euzWi8YY\nE1vCFr+qhkTkVmAB4AMeVdWVIvK16PyHgFHAEyKiwErghm4sc0LBsHX1GGNMPMl09aCq84H5HaY9\n1O7vd4HhCZ7jLeCtwy7hEQhFInbmrjHGxOHJZnE4bF09xhgTjyeDPxiJELCuHmOMicmT6Whn7hpj\nTHyeC35VjV6W2YLfGGNi8VzwhyMKYKN6jDEmDs+lY6gt+K3Fb4wxsXg3+K2rxxhjYvJe8IfdteDs\nRizGGBOb59IxGHYtfjuByxhjYvNc8Lce3PVZi98YY2LyXDoGW7t6rMVvjDExeS74Ww/uWlePMcbE\n5rngD0dci9+6eowxJjbPpWPbwV0bzmmMMTF5LvhDYTtz1xhjOuO5dAxFWsfxW4vfGGNi8WDw2yUb\njDGmM54L/qCduWuMMZ1KKh1FZJqIrBWRDSJyR4z5BSLyvIgsF5EPRGRMdPpAEXlTRFaJyEoRub2r\nK9BR2Fr8xhjTqYTBLyI+4AFgOlAKzBKR0g6L3QksVdVxwLXA/dHpIeDbqloKTAZuibFul2o7uGt9\n/MYYE1MyLf5JwAZV3aiqLcAc4NIOy5QCbwCo6hpgsIj0U9XtqvphdHotsBoo7rLSx9Da1WO3XjTG\nmNiSScdioLzd4woODe9lwBUAIjIJOAkoab+AiAwGJgDvH1lRk3PgWj3W4jfGmFi6qll8D5AvIkuB\nbwBLgHDrTBHJBp4FvqmqNbGeQERmi8giEVlUVVV1xAUJ2iUbjDGmU/4klqkEBrZ7XBKd1iYa5tcD\niIgAm4CN0ccBXOj/UVWfi/ciqvow8DBAWVmZJl+Fg9n1+I0xpnPJpONCYJiIDBGRVGAmMK/9AiKS\nH50HcCPwtqrWRDcCvwNWq+q9XVnweFoP7lpXjzHGxJawxa+qIRG5FVgA+IBHVXWliHwtOv8hYBTw\nhIgosBK4Ibr6FOAa4KNoNxDAnao6v4vr0ebA1TmtxW+MMbEk09VDNKjnd5j2ULu/3wWGx1jvHeAT\nbXq3XbLB+viNMSYmzzWLD1yd03NVM8aYLuG5dGy7Hr+1+I0xJibPBX/Qztw1xphOeS74W0f12MFd\nY4yJzXPp2NrVYw1+Y4yJzXPBH4woAZ/gTiEwxhjTkeeCPxSO2Fm7xhjTCc8lZCiidmDXGGM64b3g\nD6udvGWMMZ3wXvBHIvhtRI8xxsTluYQMha2rxxhjOuO94I9YV48xxnTGc8EfDEfsOj3GGNMJzyVk\nOKJ2LX5jjOmE54I/GFY7uGuMMZ3wXEKGIhG7364xxnTCc8FvXT3GGNM5zwW/Hdw1xpjOeS4h7cxd\nY4zpXFLBLyLTRGStiGwQkTtizC8QkedFZLmIfCAiY5Jdt6uFrKvHGGM6lTD4RcQHPABMB0qBWSJS\n2mGxO4GlqjoOuBa4/zDW7VLu4K7ndmSMMabLJJOQk4ANqrpRVVuAOcClHZYpBd4AUNU1wGAR6Zfk\nul3KLtlgjDGdSyb4i4Hydo8rotPaWwZcASAik4CTgJIk1yW63mwRWSQii6qqqpIrfQx2yQZjjOlc\nV/WJ3APki8hS4BvAEiB8OE+gqg+rapmqlhUWFh5xQexGLMYY0zl/EstUAgPbPS6JTmujqjXA9QDi\n7nm4CdgIZCRat6sFbVSPMcZ0Kpmm8UJgmIgMEZFUYCYwr/0CIpIfnQdwI/B2dGOQcN2uFrY7cBlj\nTKcStvhVNSQitwILAB/wqKquFJGvRec/BIwCnhARBVYCN3S2bvdUxbEbsRhjTOeS6epBVecD8ztM\ne6jd3+8Cw5NdtzsFw0rAWvzGGBOX55rG7lo9nquWMcZ0Gc8lZDBsV+c0xpjOeC74bRy/McZ0zlPB\nr6rW1WOMMQl4KiFDEQWwg7vGGNMJbwV/2AW/Dec0xpj4PJWQoUgEwA7uGmNMJ7wV/NEWv12P3xhj\n4vNU8AejLX7r6jHGmPg8lZBhO7hrjDEJeSr4ravHGGMS81TwB8OtB3c9VS1jjOlSnkrI1q4eO3PX\nGGPi81TwB1vH8VtXjzHGxOWp4G8dx2+3XjTGmPg8lZBtLX7r6jHGmLg8FfxtffzW4jfGmLg8lZCh\ncOsJXNbiN8aYeJIKfhGZJiJrRWSDiNwRY36eiPxFRJaJyEoRub7dvG9Fp60QkadFJL0rK9BesPUE\nLgt+Y4yJK2Hwi4gPeACYDpQCs0SktMNitwCrVHU8cA7wKxFJFZFi4DagTFXH4G64PrMLy3+QcPTg\nrl2P3xhj4ksmIScBG1R1o6q2AHOASzsso0COiAiQDewFQtF5fiBDRPxAJrCtS0oegw3nNMaYxJIJ\n/mKgvN3jiui09n4DjMKF+kfA7aoaUdVK4JfAVmA7sF9VXz3qUsfReskGO3PXGGPi66qEvABYChQB\npwK/EZFcESnA7R0Mic7LEpGrYz2BiMwWkUUisqiqquqIChFq6+qxFr8xxsSTTPBXAgPbPS6JTmvv\neuA5dTYAm4CRwHnAJlWtUtUg8BzwqVgvoqoPq2qZqpYVFhYebj2A9i1+C35jjIknmeBfCAwTkSEi\nkoo7ODuvwzJbgakAItIPGAFsjE6fLCKZ0f7/qcDqrip8RyG7Hr8xxiTkT7SAqoZE5FZgAW5UzqOq\nulJEvhad/xDwE+BxEfkIEODfVXU3sFtE5gIf4g72LgEe7p6qHLjZuh3cNcaY+BIGP4Cqzgfmd5j2\nULu/twHnx1n3LuCuoyhj0kI2qscYYxLyVJ9IMGxdPcYYk4inEjJsXT3GGJOQp4I/ZDdiMcaYhDwV\n/G23XrRLNhhjTFyeSshwRBGBFOvqMcaYuDwV/MGwWmvfGGMS8FRKhsIR6983xpgEvBX8EbXr9Bhj\nTAIeC/6IXZnTGGMS8FRKhsJqY/iNMSYBbwV/xILfGGMS8VbwhyN2uQZjjEnAUykZjKiN6jHGmAQ8\nFfxhG8dvjDEJeSolQ5GIDec0xpgEPBX8wbDabReNMSYBTwV/OKJ2cNcYYxLwVEoGw9bVY4wxiSQV\n/CIyTUTWisgGEbkjxvw8EfmLiCwTkZUicn27efkiMldE1ojIahE5sysr0F4oYl09xhiTSMLgFxEf\n8AAwHSgFZolIaYfFbgFWqep44BzgVyKSGp13P/CKqo4ExgOru6jsh3AncHlqJ8YYY7pcMik5Cdig\nqhtVtQWYA1zaYRkFckREgGxgLxASkTzgbOB3AKraoqrVXVb6DkLhiJ25a4wxCSQT/MVAebvHFdFp\n7f0GGAVsAz4CblfVCDAEqAIeE5ElIvJbEck6+mLHFgrbCVzGGJNIV/WLXAAsBYqAU4HfiEgu4Acm\nAg+q6gSgHjjkGAGAiMwWkUUisqiqquqIChGK2CUbjDEmkWRSshIY2O5xSXRae9cDz6mzAdgEjMTt\nHVSo6vvR5ebiNgSHUNWHVbVMVcsKCwsPpw5t7CJtxhiTWDLBvxAYJiJDogdsZwLzOiyzFZgKICL9\ngBHARlXdAZSLyIjoclOBVV1S8hjcZZmtxW+MMZ3xJ1pAVUMiciuwAPABj6rqShH5WnT+Q8BPgMdF\n5CNAgH9X1d3Rp/gG8MfoRmMjbu+gW7gbsViL3xhjOpMw+AFUdT4wv8O0h9r9vQ04P866S4Gyoyhj\n0kJhu/WiMcYk4ql+kWDYbr1ojDGJeColw3Zw1xhjEvJU8J8/uj+lRbnHuhjGGHNcS6qP/0TxXzNO\nPdZFMMaY456nWvzGGGMSs+A3xpgexoLfGGN6GAt+Y4zpYSz4jTGmh7HgN8aYHsaC3xhjehgLfmOM\n6WFEVY91GQ4hIlXAliNcvQ+wO+FS3tIT6ww9s949sc7QM+t9uHU+SVWTupnJcRn8R0NEFqnqJ3I1\n0ONFT6wz9Mx698Q6Q8+sd3fW2bp6jDGmh7HgN8aYHsaLwf/wsS7AMdAT6ww9s949sc7QM+vdbXX2\nXB+/McaYznmxxW+MMaYTngl+EZkmImtFZIOI3HGsy9NdRGSgiLwpIqtEZKWI3B6d3ktE/ioi66P/\nFxzrsnY1EfGJyBIReSn6uCfUOV9E5orIGhFZLSJner3eIvKt6Hd7hYg8LSLpXqyziDwqIrtEZEW7\naXHrKSLfi+bbWhG54Ghe2xPBLyI+4AFgOlAKzBKR0mNbqm4TAr6tqqXAZOCWaF3vAF5X1WHA69HH\nXnM7sLrd455Q5/uBV1R1JDAeV3/P1ltEioHbgDJVHQP4gJl4s86PA9M6TItZz+hvfCYwOrrO/0Rz\n74h4IviBScAGVd2oqi3AHODSY1ymbqGq21X1w+jftbggKMbV94noYk8Alx2bEnYPESkBLgJ+226y\n1+ucB5wN/A5AVVtUtRqP1xt3Z8AMEfEDmcA2PFhnVX0b2Nthcrx6XgrMUdVmVd0EbMDl3hHxSvAX\nA+XtHldEp3maiAwGJgDvA/1UdXt01g6g3zEqVne5D/guEGk3zet1HgJUAY9Fu7h+KyJZeLjeqloJ\n/BLYCmwH9qvqq3i4zh3Eq2eXZpxXgr/HEZFs4Fngm6pa036euqFanhmuJSIXA7tUdXG8ZbxW5yg/\nMBF4UFUnAPV06OLwWr2jfdqX4jZ6RUCWiFzdfhmv1Tme7qynV4K/EhjY7nFJdJoniUgAF/p/VNXn\nopN3isiA6PwBwK5jVb5uMAW4REQ247rxPisif8DbdQbXqqtQ1fejj+fiNgRervd5wCZVrVLVIPAc\n8Cm8Xef24tWzSzPOK8G/EBgmIkNEJBV3EGTeMS5TtxARwfX5rlbVe9vNmgd8Jfr3V4AXP+mydRdV\n/Z6qlqjqYNxn+4aqXo2H6wygqjuAchEZEZ00FViFt+u9FZgsIpnR7/pU3HEsL9e5vXj1nAfMFJE0\nERkCDAM+OOJXUVVP/AMuBNYBHwPfP9bl6cZ6fhq3+7ccWBr9dyHQGzcKYD3wGtDrWJe1m+p/DvBS\n9G/P1xk4FVgU/bxfAAq8Xm/gR8AaYAXweyDNi3UGnsYdxwji9u5u6KyewPej+bYWmH40r21n7hpj\nTA/jla4eY4wxSbLgN8aYHsaC3xhjehgLfmOM6WEs+I0xpoex4DfGmB7Ggt8YY3oYC35jjOlh/j9K\nyTug/apmqQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2521060a6a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 9792/10000 [============================>.] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.9811\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Deep CNN - 1\n",
    "- Deeper network with 6 convolutional layers and 3 pooling layers\n",
    "- Note that two (3x3) convolutional layers are stacked\n",
    "    - It is equivalent to using one (5X5) convolutional layer, but with fewer parameters\n",
    "    - Similar to the structured used in VGGnet\n",
    "\n",
    "<img src=\"http://www.hirokatsukataoka.net/research/cnnfeatureevaluation/cnnarchitecture.jpg\" style=\"width: 600px\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def deep_cnn():\n",
    "    model = Sequential()\n",
    "    \n",
    "    model.add(Conv2D(input_shape = (X_train.shape[1], X_train.shape[2], X_train.shape[3]), filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same'))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same'))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size = (2,2)))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same'))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same'))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size = (2,2)))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same'))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same'))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size = (2,2)))\n",
    "    \n",
    "    # prior layer should be flattend to be connected to dense layers\n",
    "    model.add(Flatten())\n",
    "    # dense layer with 50 neurons\n",
    "    model.add(Dense(50, activation = 'relu'))\n",
    "    # final layer with 10 neurons to classify the instances\n",
    "    model.add(Dense(10, activation = 'softmax'))\n",
    "    \n",
    "    adam = optimizers.Adam(lr = 0.001)\n",
    "    model.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['accuracy'])\n",
    "    \n",
    "    return model    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = deep_cnn()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_22 (Conv2D)           (None, 28, 28, 50)        500       \n",
      "_________________________________________________________________\n",
      "activation_4133 (Activation) (None, 28, 28, 50)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_23 (Conv2D)           (None, 28, 28, 50)        22550     \n",
      "_________________________________________________________________\n",
      "activation_4134 (Activation) (None, 28, 28, 50)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_13 (MaxPooling (None, 14, 14, 50)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_24 (Conv2D)           (None, 14, 14, 50)        22550     \n",
      "_________________________________________________________________\n",
      "activation_4135 (Activation) (None, 14, 14, 50)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_25 (Conv2D)           (None, 14, 14, 50)        22550     \n",
      "_________________________________________________________________\n",
      "activation_4136 (Activation) (None, 14, 14, 50)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_14 (MaxPooling (None, 7, 7, 50)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_26 (Conv2D)           (None, 7, 7, 50)          22550     \n",
      "_________________________________________________________________\n",
      "activation_4137 (Activation) (None, 7, 7, 50)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_27 (Conv2D)           (None, 7, 7, 50)          22550     \n",
      "_________________________________________________________________\n",
      "activation_4138 (Activation) (None, 7, 7, 50)          0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_15 (MaxPooling (None, 3, 3, 50)          0         \n",
      "_________________________________________________________________\n",
      "flatten_9 (Flatten)          (None, 450)               0         \n",
      "_________________________________________________________________\n",
      "dense_17 (Dense)             (None, 50)                22550     \n",
      "_________________________________________________________________\n",
      "dense_18 (Dense)             (None, 10)                510       \n",
      "=================================================================\n",
      "Total params: 136,310\n",
      "Trainable params: 136,310\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 19min 39s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "history = model.fit(X_train, y_train, batch_size = 50, validation_split = 0.2, epochs = 100, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnWd4lMfZts/ZKmnVG0ISSALTewfjGlzAcdxiY9wdd8c1\nxXHsfIlT7MR537xOnMRx78YFY+OKjRtuMaaDaKILVAD1Lm2d78ds164KCAFizuPQsbtPnV3tXs/9\nXHPPPUJKiUaj0WiOHwxHugEajUaj6V208Gs0Gs1xhhZ+jUajOc7Qwq/RaDTHGVr4NRqN5jhDC79G\no9EcZ2jh12g0muMMLfwajUZznKGFX6PRaI4zTJ1tIIR4DjgXqJBSjo6wXgCPAucALcC1Uso13nWz\nveuMwDNSyoe70qj09HSZn5/f1feg0Wg0xz2rV6+uklJmdGXbToUfeAH4N/BSlPVzgCHev2nA48A0\nIYQReAw4EygFVgoh3pNSbu7shPn5+axataoLTdNoNBoNgBBiT1e37dTqkVJ+DdR0sMn5wEtS8T2Q\nLIToD0wFdkgpd0kpHcDr3m01Go1GcwTpCY8/BygJel3qXRZteUSEEDcJIVYJIVZVVlb2QLM0Go1G\nE4mjpnNXSvmUlHKylHJyRkaXbCqNRqPRHARd8fg7owwYEPQ617vMHGX5QeF0OiktLaWtre1gD6EJ\nIiYmhtzcXMxm85Fuikaj6WV6QvjfA24XQryO6tytl1LuE0JUAkOEEAUowZ8HXH6wJyktLSUhIYH8\n/HxUIpHmYJFSUl1dTWlpKQUFBUe6ORqNppfpSjrna8BpQLoQohR4ABXNI6V8AliMSuXcgUrn/Il3\nnUsIcTuwBJXO+ZyUctPBNrStrU2Lfg8hhCAtLQ3dl6LRHJ90KvxSyss6WS+B26KsW4y6MPQIWvR7\nDv1ZajTHL0dN565Gc6zj8fTMNKZr9taycHUpDW3OHjmeRhOOFv4uUldXx3/+859u73fOOedQV1fX\n4Ta/+93v+Oyzzw62accFUkq+KDpAfWt0MaxvdfLkVzv5ouhAl4637UAj//5iO3//dBttTnfUbdfu\nrWX2P77m4437o26zZNN+xv/xE575Zle7dR6PxN2Fi4LL7eGRT7by48e/45dvrmfaQ5/zq4Xr2VRe\n3+m+3WHr/kZum7+Gf3y2jboWR48eW3NsII7GydYnT54sw0fubtmyhREjRhyhFkFxcTHnnnsuGzdu\nDFnucrkwmXqij7z36egzrW6yY7OaiDEbu3Ss+hYnZXWtDMqwdbrPgYY2/vXFdr7aVsk/501gwsCU\nDrdvaHPyywXr+WTzAWaekMbL103DYAhYVdVNdp77725e+m4PjXYXRoPg8SsmctaoLP82ZXWtfLej\nitLaVkprW1mzt5bdVc3+9ZPyUnjqqkmkxVtDzi2l5ML/fMe6EnXxvm5mAb+eMxyLyeBf/+y3u3lo\n8RZizUbanG5euX4aJ56QDsCOiiaue2Ele2taMAgwGw38aFw2f75wjP8YAOV1rdz1+lpWFtdyyaRc\n5k4ZwFurS3lvfTkOl4e/XTKOCyZEHgZT2+zg1RV7aWxz4XJ7SIgxc+MpBcRZQr+XLreHJ7/exaOf\nbcdiMtBkd2GzGLlqRj7TBqUSZzYSZzGRnmAhMyEGoyGyHWh3uTEbDCH/A82RRwixWko5uUvbauHv\nGvPmzePdd99l2LBhmM1mYmJiSElJoaioiG3btnHBBRdQUlJCW1sbd911FzfddBMQKD/R1NTEnDlz\nOOmkk/juu+/Iycnh3XffJTY2lmuvvZZzzz2Xiy++mPz8fK655href/99nE4nb775JsOHD6eyspLL\nL7+c8vJyZsyYwaeffsrq1atJTk2lzeEhPqZrFx+PlDTbXdgsJrZuLYr4mS4tquCn89cQazFy5bSB\nXDkjj8yEmIjHk1Ly5qpSHlq8hfpWJ0LAwNQ40mwWalucVDfZkRJG5SQybkAyLrfkle/34PZIkuPM\n2F0eXrtxOqNzkvzHbGxz4vZIzEYDe6pbuO3VNeytaWH2qCw+3LCPe84exm2nnwDAxrJ6rn5uBbUt\nDs4Z05/rZubzpw+2sLm8gWevncz0QWk8881uHv18G21OD0JA/8QYhvRL4MyR/ThzZD/W7Knl7jfW\n0S8xhud/MoXBGfH+tny0YR+3zl/DgxeMZkdFEy98V8yYnCRmDE4jMcbE7qoW3lpTyuxRWTx44Wjm\nPfU9tc0O3r/jJGqaHVz93AoMQnDFtIF4pORAQxsLVpVyytAMnrhyIrFmI++tL+f/vbMRj0fy0IVj\nQgS+vsXJza+s4vtdNfzmnBHceMqgkM+/zenm8qe/Z83eOiwmA2aDoNnh5saTC/jND0f6t2toc3L1\nsytYV1LHD8f254/njaKi0c5jS3fw4YZ9hMuA2SjITo4lP83G0H7xDOmXQEOrk6+3V7F8VzVZSTHM\nv2EauSlxEb8XFY1trNhdw7ljs6N8EzU9TZ8X/j+8v4nN5Q09es6R2Yk88KNRUdcHR/xffvklP/zh\nD9m4caM/HbKmpobU1FRaW1uZMmUKX331FWlpaSHCf8IJJ7Bq1SrGjx/P3LlzOe+887jyyivbCf8v\nfvELfnrb7fz9n/9iw/p1vPj8c9x+++3k5ORw33338fHHHzNnzhwqKipoEnE0tjnJT7ORGBs9J19K\nSV2rkwMNbThcHmLMRtoq9zJ2dOh7Xri6lHvfKmR4VgL9k2L4vKgCk0EwZ3R/Lpmcy4mD0zEaBG1O\nN4Wl9fzfJ1tZvruGKfkpXDEtj+LqZrYfaKKm2UFavIU0mwW3lGworWfzvgZcHsmF43O4+4yhGI2C\nuU8so9nhYv4N0zjQ0MaL3+3hq22h2UaZCVYeu2Iik/NSuOO1tXy0cT8Lbp6OR8J1z68kMdbMs9dO\nZnhWIqDEct7T37O7qomBqXFsO9DE2aP68cuzhpGXZguJtH2s2VvLjS+uwuWRPHvNZCbnp+J0ezjr\n719jMgg+uutkTEYDizfs4+GPiqhobKPN6QHgplMG8evZwzEYBDsqmjj/398yIDWOsrpWEmPMvHLD\nNArSbf5zvbFyL/e9vYHxA5LpnxzLh4X7mDgwmUfmjic/aDsfdpebn7+xng837OPaE/O55+xh2Kwm\npJTc+fo63l9fzn+umMg5Y/oDcN/bhSxYVcriO09mWFYCUkruen0dH27Yx98vHc9540LFeF99K/vq\n22h1uGm2u6hsslNa20pJTQu7KpvZUdmEw6Xe6wmZ8cwYlMY768pIibOw4OYZZCWFBgVOt4dLnljG\nupI6Pr77ZP//RXN46Y7wH5sexVHA1KlTQ3Lg//nPf7Jo0SIASkpK2L59O2lpaSH7FBQUMH78eAAm\nTZpEcXFxu+NKCdNOn8OWfQ1kDR7Jwrfexu5y8+233/qPP3v2bFJSUqhvcdJidGI0CEprWxlqMWIy\nKlFraHWyv0ENdjMIgdsjsbvcxJiN9E+KpbLRTkWjnWe+2cWEgclUNTlYs7eWJ7/axUknpPPEVZOI\nt5rYXdXMC//dzaK1Zby3vpzspBhS4y0U7WvE5ZEkxJj4y0VjuHTygE5v/e0uN01trhA75bUbp3Pp\nU8v44T+/BZTI33b6YNJsVlweDwLB+ROy/Xccf75oDOtL67j1lTU0tDnJTorllRumkZ0c6z9mUpyZ\nl6+fyrynvqeh1cVTV00KsX0iMXFgCot+OpNrnl/BFc8s57HLJ7KvoY3dVc08c/Vk/+d6zpj+foG1\nu9w4XMpa8XFCZjx/u2Qct85fQ0G6jVdumEZOUNsALp0ykKRYM3e+to7C0nruOXsYN58yyH+OcKwm\nI/+8bAIZCVZe+K6YDwrLuf30E6hssvP++nJ+NXuYv00A95w9nI827ue3727kjZum89Ya9b/75VlD\n24k+QP+kWPonxbZb7sPl9rC3poVYi9G/3UUTc7jq2RVc/vT3vH7z9JA7wkc/2866kjqEgHfWlvPr\nOVr4jzaOSeHvKDLvLWy2QGT25Zdf8tlnn7Fs2TLi4uI47bTTIo4wtloDgmc0GmltbfW/drg8FFc1\n4/J4cGAgO85Mbko8bpeLstrWdscC2N/YRlZmIjnJseyobKK0tpW8tDjqW52U1LRiMRmwmgx4pMRg\nFPRLjCMp1owQgpQ4MzVlBh58d0vIMc8fn83/XjzOHxUXpNv4w/mjue+cEXy25QBvrynD4fJw0ymD\nGDcgmWkFqSTHWbr0mVlNRqzxof7/wLQ4Xr1xOv/6fDuzRvTjrFH9MEcRQIDEGDP/umwiFz/+HUP6\nJfDSdVPJSLC22y493sqHd56EQESM8CMxMC2OhbfM4LoXVnLzK6uJsxiZmp/KrBGZ0d+PqX1/xpwx\n/XnzlhmckBFPii3yZzN7dH/euc2G2SgY0i+h07YZDYLfnzeKH43rz/98vJXfv6+K3F4yKZdbTx0c\nsm2qzcK9s4dz39sbePTz7Tz19S6mFaRy62kndHqeSJiMBgYF2V8AEwam8PxPpnDNcyv48ePf8btz\nR3HGiEy+31XDY1/u4JJJuVQ12XlvXRm/OnuY7g84yjgmhf9IkJCQQGNjY8R19fX1pKSkEBcXR1FR\nEd9//323jt3qcLOvvpVmh+qYHJqZQGZKHPtjTP5OuMnTprNgwQLuvfdePvnkE2pra/FIyE2JJcZs\nJCvRyr76NkpqWqlvdRBnMZGfHofREFn0TEYDafFWXrl+Gi6Ph/R4K+nx1na37T5izEbOHZt9WDzb\ngnQbj1w6vsvbjx+QzGc/P5XMRGu7DsxgIolyZ6TFW3n1xuncOn8N32yv5N45ww9qzMOU/NROtxmZ\n3f1IeFJeKq/fNJ1vtlexZm8tPz3thIjtu3TyAF5fWcI/PttOUqyZv186Pmpn7cEyJT+Vl6+fyq8W\nFnLjS6s4cXAauyqbyU+z8fvzRvHZlgPc9fo6VhTXMH1QWucH1PQaWvi7SFpaGjNnzmT06NHExsbS\nr18//7rZs2fzxBNPMGLECIYNG8b06dMBcHk8uD2SfXWtSGfktLkWu4s2p5s4q4lh/RIwCBESHZmN\nBuIsJq657R4euPsmXnr5ZSZMmkp6Zj/ys9L8GTTp8VYa2lzUtTqIt5rIT7N1Kco6aUj6oXwsR4xI\nXnhPYbOaeO6ayeyrb2NAauTOyyOJEIJThmZwytDoxQwNBsGD54/mxpdW8cfzR4VYYT3JpLxUPr77\nFF5dvpe/f7aNZruLt6+eic1q4syR/YizGHlnbZkW/qOMY7Jz91jAIyW7q5ppsbsRQr0GyEqMITNR\nRdVSSnZVNmN3eRiWlRA1Imt1uNlcUo3FYkIKA+tXr+Avv/klGzesxxAU7TndHupbnaTaLCHLo3Gs\nfaaa7iOl7LVR2vWtKosr2Bb6+Rvr+HTLAVb+5owupwYDVDXZsbs8xJqNxFmM3dr3eEV37h5hpJSU\n1rTSbHcxIFX56m0ON5VNdvY3tGE1G0mKNdPY5qLZ4SInObbD2/BYixFnQwVXXX0FAkms1cqLzz/b\nTtzNRgPp8e39bs3xS2+W5kiKNZMUlll2wYQc3l5bxpdbK5g9un+UPUMpqWlh1iNf+TOJAB68YDRX\nTs/r0fYez2jh7wGa7C7K61oxGQSxFiNut6Su1UFWUgwp3o7POKuJAWYjTncTpTUtWDPj2VffhtVk\nJDVKB2Aw0yeMZvOG9Yf7rWg0PcqJg9NIj7eyaG1ZO+FftrOaBz/czF9/PDZkHMdTX+9CSsmDF4zG\n5fYwf/leXlpWzBXTBuoaUz2ELtlwiDTbXRRXNeOREreUVDU5qGlxkGazkBEWfRsMgoGpNoRQ+d52\nl5uspBj9Zdb0WUxGA+eNy2ZpUWXI+IxVxTVc/+JKNpU38MB7m/BZzlVNdhasKuGiCblcOT2Pa2cW\ncPWJ+Ww70MSmHh67czyjhb8bON0eKhraaGh14nJ7aLa72F3VjNloYHBGPEMyExjVP5FhWQlkJ8dG\nFHSLycDA1DikBJvFRGIXR9xqNEc9G9+Cly8ifBjwDScXkJcWxzXPreB3725k2c5qrn1+JVmJMdxz\n9jBW76nlvfXlALzw32Icbg83nRoYofyjsf0xGwWL1h70PE6aMLTwdxEpJSU1LexvaKO4upnN+xrY\nValEf1CGzZ97bjAIrCZjh1F8fIyJwZk28tLidLSv6TtsfBt2fg7VO0IWZyfH8v4dJ3H9SQW8tGwP\nlz39Pak2C6/eOJ1bTh3MyP6J/PWjIiob7by0rJjZo7JCymYkx1mYNbwf764rw+VWvr+Ukr8s3uK/\nYGi6hxb+LlLV5KDJ7iI7OZZBGfFkJaoRrMGi3x3iLKaoIzU1mmOS8nXqcc937VbFmI389tyRvHrj\nNM4fn82rN04jK0kVgnvgRyMpr29j3lPLaGhzcUvYgDRQI4Wrmhx8s70KgBe/K+bJr3fx2Bc72m2r\n6RytPF2g1eFif0MbSbFm0mwW4q0mMhNjyEmOjSr68fEqYikvL+fiiy+OuM1pp51GeNpqOP/4xz9o\naWnxv+5KmWeNptdpqoSGUvV877Kom504OJ1H500IKe42bVAa54zJYmdlMycOTmPcgOR2+502LJOU\nODNvry1jQ2k9f15cREKMia0HGimrizyyXRMdLfwRkFLS4nDR4nDRbHext0Zl7ORE8e07Ijs7m4UL\nFx50W8KFf/HixSQnt/9haDRHlH3eaD8+K2LE3xn3zRnB0H7x/PzMoRHXW0yqnPUnm/Zz26trSI+3\n8Py1UwBVTVbTPbTwhyGlZG9NCzsqmthR0cTOSpV98/T//Yknn3jcv93vf/97HnzwQWbNmsXEiRMZ\nM2YM7777brvjFRcXM3r0aABaW1uZN28eI0aM4MILLwyp1XPrrbcyefJkRo0axQMPPACowm/l5eWc\nfvrpnH766YAq81xVpW53H3nkEUaPHs3o0aP5xz/+4T/fiBEjuPHGGxk1ahRnnXVWyHk0msNC+Vr1\nOOV6qNsDDd3z3gekxvHJz05lcgelLi6amIvd5aGsrpV/XT6BSXkpDEiN1cJ/EBybKSUf/Rr2b+jZ\nY2aNgTkPU9XkoL7VSUaCFZu3DozFZOCqKy7n7rvv5rbb1PTCCxYsYMmSJdx5550kJiZSVVXF9OnT\nOe+886LeFTz++OPExcWxZcsWCgsLmThxon/dQw89RGpqKm63m1mzZlFYWMidd97JI488wtKlS0lP\nDy2tsHr1ap5//nmWL1+OlJJp06Zx6qmnkpKSwvbt23nttdd4+umnmTt3Lm+99RZXXnllz35eRxKP\nGz75fzDiPMibcaRbowHl76cNgSFnwtKHVNQ/JrLFebCMy03iook5TMlPZVKeukD8YFgmb6wqoc3p\n1qN7u4GO+INotrvYX6+8/KzEGBJjzSTGmokxG5kwYQIVFRWUl5ezfv16UlJSyMrK4v7772fs2LGc\nccYZlJWVceBA9Gn/vv76a78Ajx07lrFjx/rXLViwgIkTJzJhwgQ2bdrE5s2bO2zrt99+y4UXXojN\nZiM+Pp6LLrqIb775Buha+edjmlXPwff/gVXPHumWaHyUr4XsCdBvDFjiO/T5DxYhBI/MHc9lUwf6\nl50+PJM2p4dlu6p7/Hx9mWMz4p/zcI8f0un2sLeiCYtJkJsS2cu/5JJLWLhwIfv37+fSSy9l/vz5\nVFZWsnr1asxmM/n5+RHLMXfG7t27+dvf/sbKlStJSUnh2muvPajj+Oio/PMhU75O/bDTD67Eb6e4\nHColcPAsMEUY0dxQDp/9QT0vWdHxsSqKQAjIGNbz7ewIlwM+/4O6K22phtZaOO0+mHhVz57H7YL9\n66H4W2jYB2f+AUxHoGRH4wFoLFfCbzTBgKmwp+eFPxLTB6URYzawtKiC04dFLp+taU+XIn4hxGwh\nxFYhxA4hxK8jrE8RQiwSQhQKIVYIIUYHrbtLCLFRCLFJCHF3Tza+p3C4POyuasbtkQxMtUUtZXzp\npZfy+uuvs3DhQi655BLq6+vJzMzEbDazdOlS9uzZ0+F5TjnlFF599VUANm7cSGFhIQANDQ3YbDaS\nkpI4cOAAH330kX+faOWgTz75ZN555x1aWlpobm5m0aJFnHzyyQf7EXSNFU/D0z+AF86B5sMQYdXu\ngednw2vz4Ov/jbzNR/eCxwlTblBecmOUOyy3C+ZfDK/OVdZQT+FxQ+GbsOQ3arDSf05Ur/3rPfDO\nLbDs3+BogqQB4HHB+td7rg2ghPV/B6n/x6e/g+WPw57/9uw5uoqvYzfbW1p74IlQsVld8A4zMWYj\nMwen80VRBUdjwcmjlU6FXwhhBB4D5gAjgcuEECPDNrsfWCelHAtcDTzq3Xc0cCMwFRgHnCuEOEyh\n4kHQXImztoQdlU043R7y0uKItUT3CUeNGkVjYyM5OTn079+fK664glWrVjFmzBheeuklhg8f3uHp\nbr31VpqamhgxYgS/+93vmDRpEgDjRg1jwvhxDB8+nMsvv5yZM2f697npppuYPXu2v3PXx8SJE7n2\n2muZOnUq06ZN44YbbmDChAlde9/SE3m52xV9+Ye/hMW/hPyZ0FIDH9zVboRmt2hrgKLFsPsbqNgC\nm9+FJ0+Gqu0wYBr891Go2RW6z9aPYMt7cOqvYOylallplKh/20dQXwK1xbD909B1Hg+4nQfX7s8e\ngLdvgJXPQHOluqN4+wb44kF13I/vVSNYz/gD3PgFXP46jL4YSleC8+Dv4trx/WNgMMPFz8Eda0AY\nYG/35oHoMcrXAgKyvNZl3gxAwt7lvXL604dnUlrbys7Kpl45X59AStnhHzADWBL0+j7gvrBtPgRO\nDnq9E+gHXAI8G7T8t8CvOjvnpEmTZDibN29ut+yQ8Hike98G6SpdK7eU18tWh6vL+/U4+zdJWbmt\n547nckjZUK6O2VwVaLPbKWVdiZRla6Vsrgr9TKt2SPlwvpQrn2t/vDevk/KBRCmX/D8p3S4pv/m7\ner3mle63ra1Byq/+V8qH89Qxgv8eP0nK6p1S1pdL+VC2lPPnBvar3iXl/42Q8t/TpHTapXS2SfnH\ndCmX/CbyeZ7/oZSPjJLyb8OlfOmC0HWLfirln/pJ+cbVUm5cJKW9uWtt37BQtfODn6vPQUrVlndu\nU8sfm+79nMLatOVDtXz3N6HLa/dKWb4ucKyu0tYo5Z8ypfzgF4Flj58k5Qvnhm7XXC3l61dIuXdF\n947fXebPlfLfUwOvHS1S/iFNyk9+e3jP66W0tkXm3fuBfPKrHb1yvqMVYJXsRFt9f13x+HOAkqDX\npcC0sG3WAxcB3wghpgJ5QC6wEXhICJEGtALnABFHLAkhbgJuAhg4cGCkTXoUp70Fs8cJAk5Is2Dq\nLCNAepS/3FID6UPAHGViCynBZVdea1dy/j1ucNvVn7M1+nG7grMNGvdBWz0gwWiBur3KDolNVn6z\nx+VdXgpub/tcdnjzWmitgR2fweSfBI7paIZNi2DqzXDWn9SyE+9QUfRHv4K8EyG1ILwlgfe2/EkV\nAfvuMmp3Kwtg6GyY/lP1GTVXqruKkeeD2TsD2Kn3wqe/ha0fQ0wivH6FOsa8+QHvv/94KFnZ/rwH\nNkPxN3DG79Vxlz6o7iTSh8DOL2DdKzBwhrJGNr8Dcelw0ZNwwhlBn2WrOk7mCLDEwf6N8O7tMGA6\nnP0XMHi/LyYLnPcv1Y/wyW9h/BVw5p9C25M3AxBQ/F/IP0ktkxJevkCVN4hJgryZMPUmGBx6ZxeR\n7Z+Aqw1GXRBYNnAGrH1Z3ckYvaWRN78DW95Xd1XXfazeSzhNFepuq/845c0fDOXrQtttjoWcib3m\n8+ckxzI8K4Evt1Zy0yntR/1q2tNTnbsPA48KIdYBG4C1gFtKuUUI8VfgE6AZWAdENFyllE8BT4Ga\niKWH2hURKSWN9dX4MoZNHgcQecpBQAljbTE4W9Qtdd0eSB+qngfjcUN9qRLQlHyITem8MS574HlL\nlfKEu4uUSjwbylWbbBlgSwOjVV0EGvdD0wGw2CBxsBKGyiJoqVTC/tnvYX8hpA5WloSUgYtW2RqQ\n7lBRNBjhwsfh8Znw4S/gqrfbt6l6J7zzUyj5HrInQpx3Bqa0E2DaLZA7qeP3NP1WWPsKvHeHulCk\n5MHlCyAt6Ic9YKrqd3A5QjuCVzwFphiYeI36n3z9P2rZmX+ED36m2nDVO+pzKP4GPr4PXrkYTvkl\nnPQzWPMyfPt3aNqv7JTcyer/ak2EuS+173QWQl0Mx8yF+Mz2F/zYFMgarc7FvYHPtXoHTPqJ+ny3\nfwpvXgM/2wzW0Plt27H5XbBlKrH3MXA6rHhSdSjneNOEty2BhP7ei8xFcP0SSB6oLoalK2DV8+qi\n7vHaXmMvVRZVYtfq5gOqU7lpv7oIBzNwBix77NCDmS4yfVAar6/ci8Pl6fIcy8czXRH+MiBYjXK9\ny/xIKRuAnwAIlQ6zG9jlXfcs8Kx33Z9RdwwHheyh2YSqmx3EuppwG8wYpVNFT0SZ/9TREig6leKN\nbGt3q0gpISuwnbNNLXd5fVxnSxeF35txY46DllpIyA5Ek13B2Qb1e5WAWxPVD9sYNBlGbLKKKN0O\nFel7Pz+ZnAe7y+DFH0HZaph+m4rcF/9SeePJ3rsun4eeGzaxT/JAJXZLH1IXxZT8wLqixbDwOiWQ\nFz6pBKW7/zejGc75H3jpfCg4Fea+2P7zHDBVdaLuLwy0r7UWCt9QOeRx3kv7qAthnepUp7YYrnk/\ncGcx6DS44XNYfI/qUP7vP9XdV95MlSVTsVlFzI4mdeFJ6EdUOlqXf7JKQ/XdDW5cqP4fZ/xe/Y9K\nVsCzZ8L612DqjdGP42hREf+4y0K/JwPVdJ/s/V4Jv6MFdn2pLn6TroHn56jPMnWwSrV0NKnvy5Tr\n1V3KlvfUe9/yAYz4kbo7yBgOBaeoO55gCheoi9iI89T3HFRGTzC5U9QF5cCm0O9O4wH44k8w+y9g\njTLJfEM5vHAuzPmrGhfQBaYPSuWF74opLK3rcBCYRtEV4V8JDBFCFKAEfx5wefAGQohkoEVK6QBu\nAL72XgwQQmRKKSuEEANRdtD0g2loTEwM1dXVpKWlHZL4tzndVNY3M1zYwZalomxXB+mOzd4a4hnD\nAqlyrcnbqBIbAAAgAElEQVQqio5JUhFhc4W3o8+gflgNZV3vyHO1AQISc6B6uxIuWxfmwfW4VRt8\n500eCLGpkQVWiJA0Pykl1U0OYkwGJfrZE5UAVWxSG5SuDAh/yQp1dxMX4cc0/nJY+mdYOx9+8JtA\nu5bcry4iV74FiYcwOfug01THZXKeShMMJ9drTZQsD4jL2vlKjKbeHNhu6s3qYrDiKRh3uRKzYCxx\ncMFjyoYp+gCm3ayEuicrp+bNVGMPylarzuuNb8GQs5Tog7qI5UyG7x+HydeDL7OsrkR9Lwb/QL3e\n/ol6f8E2D6jPOXmgEvUZP1XC7GqDYbOh3yi47A1YcLWy/sZeqt7rkDMD4tt/rLoAfPEn2P0VFHqz\nkAadDle/EziPvUldJNvqYM1LgFDfv6wxoe3xZfiUrw0V/k2LlCU1+Acw+qLIn9VHv4KanbBzaZeF\nf2qBuqNcvrtGC38X6FT4pZQuIcTtwBLACDwnpdwkhLjFu/4JYATwohBCApuA64MO8ZbX43cCt0kp\nD6rCWG5uLqWlpVRWVna+cfT3QmWTA4u7lSIaIN6gvsCyGhJaIu2gog9zDNQFZZh43NBYBSXVyneW\nHhWxx6ZAbRk0V6kIuyLyBOshNFcq372+BBrroLQu9E7CR1udiuKEQf15nKodlnh1AaqrALo+dD0m\nJobcMSeD/BsM/6GKzvuNBlMslK6C0T9W779kBQw/J/JBknLVD3jdq3Dar1UEuuV9decz9+VDE30f\naR14ton9IWmgauOM21SK6bJ/K5uhf2BwHLmTlKjW7IKzHox+vPGXqb/DQd6J+H1+t1NZb+EjW2f8\nVN0pbV8Cw+aoIODFc9Vdyum/gVPuUTZPXLpKmQxn4AwV5UupMqAs8eqCA6qf4ZfbOr6YpRaoLCFQ\n5/7+cfjqr2oUbp73fGtfUd/Faz5Qd5mb31EBR/hdQWKOshx9pRx8lHr7ZIq/jSz8RYvVdwgBlVui\ntzW86TYLQ/vFs3x3Dbd1oZvkeKdLHr+UcjGwOGzZE0HPlwERqytJKXskudxsNlNQEKUTsYs88uk2\n/vn5br474WWy69bAz4tUdFH4Bvx6b/sfxd7lsODH6scw4qTQdZt2qB/piB+pjsh+QRmuXz6s/u4v\nD/1BlK1W/nJMYJo5Hp2nbpMveQFWfgcf/hyu/wwGTAlss/Qv8NXDKvoyWdXFwpoIs34LOZ145Z0R\nbCsYzaotvoFR1TtVf0VuB51+E66EhT9RgjP4ByoNM3WQupj0BgOmKmHy5c+3VMNlEXLmL31F3dnZ\n0nqnXeHEparIu/gb1UdkiVcd3MGMOE8J5rLH1N3AWzdCfRkMOVtZak0VyrcfOzfyHdDA6eq7XLNL\nbTf49NABXd25g4lNgZl3q36ALx+Ga95TfQPLHlMd3AXen/Ww2ZH3F0J9l3ylmn34rMNIYw7sjcpq\nzBypbKZupqdOK0jj7TWluNweXfK8E46bT2f1nloeW7qDSyZkkl35X/XDMhiUhWNviFxUautiMJhC\nOzZ9jLpACfvcF0NFH7wjRaW6Rfdhb4Rnz4Yv/xpY5mhRg5YyvPn/Y+cqQfjoHtj+mRKz5U8p0R9/\nJVy1CC5/Q+WHX/3OoYt+JAZMUZ65yx74kQ4IT+IKYvgPlUisfUX9mMvXwIzbu9dPcUjtnapGjS65\nT9kgZ/85YDMEk9hfXZCOJPknqYvqlvdg+LntOz2NZpXZU/wNvHUD7PhU+dyXva4yoFY+Dc7m9jaP\nD19n78pn1GcSfmHpLpY4mHmnsn72fq+i+/q9allX6D9eRe0O79104wFlNSVkq+SCprC796V/Vr/D\nHz2qrKPGcmgNMwgqiqIODJtakEqzw81GPUVjpxwXwt9sd/HzBevonxTDH8Y1KKEfNket9KW4VRa1\n33HrR+rHGhyhB+PrIAzHJ+SV2wLLytYoe2bnF4Fl1dsBGSgpYE2Ac/6morz5P4Z/jlN3JMPOUT+G\n3pitK3eKsqn2rVfeeUyS8vijYbIqz7joA/jiIWVDjL88+vaHo70Ay59Qd19Tbui9c3eXvJnqrqOt\nHsZcEnmbSdco23DT2+puavJ1KkA5+88q4+aEMyDvpMj7pg9T/6+Vz6jXQ8469DZPvk79T798WN3N\npQ2BoXO6tm/2BGWD+goq+myeGarQYUjUX7VD/Q8nX6cu5r7fZVXQb8jtVB3gH7UrHgDAtEHK21+u\n6/Z0ynEh/A9/VMTemhYemTueuOLPVZpjwalqpV+kt4buVL0TqrYq0e0uqYNBGEMvJr4vfeUW1SkL\nKnoByAjKrx5/GfxsE/z4WUjMVT/ei5+LfGt/OPAJaelKlSOfOyXQ0RiNCVeqi8Xe71TE2gvpe36y\nxiihTB4I5/27dy6OB4vPb49Lg0GnRt4mNkWNTB46B875v8D7EQJOult1mEf7LhgMyoZxO9TdYHwP\n1K6x2FSEv2upuhM88fbOvw8+fHdevpIOpStVMsSka8BsUz6/jzUvqL6rU73prr7fZUWQz39gowra\nij5QaaJhZCbEMCjdxvLdNd17j8chfV74pZR8UFjO+eOymVqQqqKMgdMCudK2dPVDDO9I2urt0hjW\nxegmGJNF2Qrhwm/2+v27vlKPlUXKSgq3IEwW1fF33UdwxYLeFdKELNVhuuNzlcrYkb/vI2uMGgBk\njus4FfFwYDTDZa+pvHxfhszRii1NWTzTbglNuQ3npJ+pUg/R7ig7wpfW2dWovCtMuUH9RmyZMHZe\n1/dL6A/x/QIdvKUrVae7NUH9Bn0Rv8uuEgSGzQmkxCbnqe9T8G/IN1jP0dS+DIeXaYNSWbm7BrdH\n1+3piD4v/KW1rdS2OFWKl8etvkhZY0M3yhjRPuLf+pEqMetLa+wuGcMCx5RSfelHnq8yIHZ9qZZX\nblV3B5GqUB5JBkxRFTKRXR/NecHjSoAjpX0ebgad1nH2z9HEvPkqoj9cDJujBHfUhT13TItN9TPM\nm9+9i1FwB6/bpexO3x1l3kwVWDRXQ9GHqlN+0rWBfQ0GZTEGR/wly9UMX3HpygqLwLSCNBrtLrbs\n0z5/R/R54S8srQdgXG6yynZwtansimAyhqkLgq/oWEuNyoc+mGjff8zh3vM51GNLteokLThFdZZJ\nqe4yMjsu7HZE8P04EV3vQO43Sgmw5siSOQJ+UdTzZbMHTD24kg79xyvLtHSF6t/wfbfyvVlBe/4L\nq19Qd5mDfhC6b+aIsLvmFepOYeT5KmvJ0dzudD6f/3vt83dI3xf+sjosRgNDs+KVRwgRhH94oLQB\nqGhfeg5d+KVbDUQp9ZYnyp2ixLGhTHV41RYHvMyjCZ+902+UqpOj0Rwsvg7eld5Jc3zCnz1BjRlZ\n+7IKhCZe3b7vIGO4qj3VWqd+m3V7VfA0+iI1iG3bx+1O1z8ploGpcfx3R9VhfmPHNn1f+EvqGd4/\nAavJqIpuCaPKfgjGF3VXFqlo/4s/qS9deP2R7pAxNHDM0pUqTTNzRCAqXvmM+kEcjcKfNUZ1vgXX\ngtFoDgZfB+/md5Xf77NOTRYVvW//RHXqTrii/b7BGXe+sSW5U9X3Mj4LNka2e340rj9Lt1aydKue\nizcafVr4PR7JxrJ6xuZ60zEPbFIDqMJ9yowg4f/gZ2qA1IVPdj17IRJpQ1CjD723uTkTVW57aoHq\nuCpcEHruowmTRRX0Ov3+I90SzbFOQpbqc/A4VbQfnHXlq1Q6dHbkUd7BmT0ly1U2Xv+x6nc06gLV\nwdvW3su/4wdDGNovnnsXFlLb3IXR88chfVr4i6ubabS7GJvjzfao2NR+sBWooeWxKaqWyuZ3lOBF\nGgTUHSxxqqJk+VpV0jc4O2bQqcrvFMajt1Mya8yR6ajV9D18BdzCC/2dcIaK9qONvUgaoO48fXfN\n2eMDI5FHXaSK6W1d3G63GLORR+aOp7bFwW/f3diDb6Tv0KeF39exOyY3SY2crS1u7++Dd17WEcpD\nHDhDDVXvCTKGq/r20h3UYUrA7kkbfGTmSNVoehO/8E9tv/yenXDCrMj7+UbW7ytUAVRw53LuFNUh\nvOr5iDPBjc5J4u4zhvJB4T7eXVfWbv3xTp8X/hizgSGZ8YHBUpkRhB9UNGFN9Fo8PVRuIGOYKsAG\nocLvHzzWy5OAazRHgjGXqKg++Dfgo7O7yozhamCg2xF64TAY4KS71HwPOz6PuOvNpwxiUl4K9yws\n5J21WvyD6dPCv6GsjlHZSapgU7SMHh8/+C3cvlLZMz2FrxM5dVBocTBbOpzyK5h4bc+dS6M5Wkkt\ngB/+38GNVwlOdw5PJ51wteos/uJPEaN+k9HA01dPZsKAZO5+Yx1/+WiLHtjlpc8Kv8vtYWNZA2Ny\nvB27FZvBkhB9QJYlLnI55EPB1zkVafTrD34DQyIUf9NoNAF85UySB7b/fZoscOqvVUmIog8i7p5q\ns/DKDdO4anoeT361izteW3OYG3xs0GeFf2dlM61ON+MGBGX09BvZu7VcMoerUYZDz+69c2o0fYnM\nDoInUAUC04aoAoGeiLO6YjYa+NMFo7lz1hAWb9jPxrL6w9TYY4deqvzV+xSWqnKuY3KS1W3ggU09\nO4y9K1hscM+Oo7twmEZzNJM0QGXwRKv4ajTB6fepuTGW3K+y85orQ4u4DZgKk67l+pMKeOrrnby2\nYi8PXTgm8vGOE/qs8O/bsY4BlhYGpdtUje+2uuj+/uFEi75Gc/AIAZc83/E2Iy+E/v9UZZ1Bib/Z\npva1N6hpLsfOIyk2hh+OyebddeXcf84IbNY+K3+d0jffuZRcue1OLjObMNRPD9T0PhLCr9FoDi8G\nA1z3sRrMFZcaWvl02xJ4da7K/hl0GpdNHcBba0r5oLCcS6ccZAHGPkDf9Pgb95HqqSHDXQEvnKtq\n74Ca0k2j0fQ9zLGqpHN4ueu8mWoOAO8ESJPyUhiSGc9rK0qOQCOPHvqm8O8rBOCDnLvBXg+rvJOa\nHO312jUaTc9ijVeF3XYuBUAIwWVTB7KupI7Nx/EUjX1U+NfjkYINGT+Cq99V09EdagkGjUZzbDL4\nNDV7WLOq2HnRxBwsJgOvr9x7ZNt1BOmbwr+/kGKywGpTw8J/+j2c968j3SqNRnMk8NX5906AlBxn\nYc7oLBatLTtuB3R1SfiFELOFEFuFEDuEEO1mOhZCpAghFgkhCoUQK4QQo4PW/UwIsUkIsVEI8ZoQ\n4iDmk+sect96NnnysBq9by8xWxcc02iOV7LHQ0yy3+4BmFqQSmObi4rGtiPYsCNHp8IvhDACjwFz\ngJHAZUKI8F7S+4F1UsqxwNXAo959c4A7gclSytGAEejGpJ0HQUsNor6EjZ4CzMa+eUOj0Wi6gcGo\nKuLuWuov7ZCdrOaxLq9rP2n78UBXlHEqsENKuUtK6QBeB84P22Yk8AWAlLIIyBdCeGdNxgTECiFM\nQBxQ3iMtj8b+DQBskvmYTVr4NRoNMOh0NfOdN7U71yv8pbVa+KORAwTnPpV6lwWzHrgIQAgxFcgD\ncqWUZcDfgL3APqBeSvlJpJMIIW4SQqwSQqyqrKzs3rsIZt96ADZ58rDoiF+j0QAMPl09eu2eQMSv\nrZ5D4WEgWQixDrgDWAu4hRApqLuDAiAbsAkhrox0ACnlU1LKyVLKyRkZGQffkv2FuOOzqSVRR/wa\njUaRkq+q5O5Swm+zmkiOM1NW13Jk23WE6IoylgEDgl7nepf5kVI2SCl/IqUcj/L4M4BdwBnAbill\npZTSCbwNnNgjLY/GvkIcmapv2WLU5RI0Go2X/JPVFI5espNiDz3ilxI+vk/NsncM0RXhXwkMEUIU\nCCEsqM7Z94I3EEIke9cB3AB8LaVsQFk804UQcUIIAcwCtvRc88NwtED1dtrSvMKvI36NRuMjPhPa\n6v0dvDkpsZQdqsffVq+mbN2+pAca2Ht0qoxSShdwO7AEJdoLpJSbhBC3CCFu8W42AtgohNiKyv65\ny7vvcmAhsAbY4D3fUz3+Lnwc2ATSQ0uaqsmjs3o0Go0fawJIDziVvZOTHHvoWT2+KqCuY2tS9y4V\naZNSLgYWhy17Iuj5MmBolH0fAB44hDZ2nX3rAGhKGQns1sKv0WgCWOLVo70RLDZykmNptLuob3WS\nFGvueN9ouLzC77b3TBt7ib6ljPsLITaF5hg1U4+2ejQajR9rgnq0NwE9lMvvi/jdzkNpWa/Tt5Rx\nXyFkjcXpVh6eTufUaDR+fMLvaASUxw8cms/vt3p0xH9kcDvVvLr9x/mFX1s9Go3GT7DVA2Qnq+ox\n5fU9EfFr4T8yGExw2wqYdjNOtwfQVo9GownC6hN+ZfWk26xYTIauRfyOZlh4PTTuD11+jHbu9h1l\nFAJSCyApF7tLCb9Z5/FrNBof1kT16I34DQZBdlIMZV3x+Cu2wMaFsOe70OXeDCHcWviPOP6IX1s9\nGo3Gh8/q8Xr8oDp4uyT8jmb16Azb1m/1aOE/4viEX3v8Go3GT1hWD3Qjl98v/GElHly6c/eoweHS\nHr9GownDHAvC4Ld6QEX8FY12v2bgdobU7ffjE3zfBcC/XEf8Rw064tdoNO0QAiwJ4AiK+FNikRL2\n16uaPXL5k/DyBVC1I3TfqFaP9viPGhw6j1+j0UTCmhAS8ef46vLXtYCUNHz3HAD7DoRNG+IX/igR\nv7Z6jjza6tFoNBGxxkcU/vK6NmTpKpKadgJQuj9sThCf4DvCPH6nt7qnjviPPAGrR6dzajSaICzx\nIVZPVpIaxFVW28r+r57xL6+sqQndzyf42uo5enG6PQgBRoMWfo1GE0SY1RNjNpKRYKWypobkne+x\nnmEA1NbWhe6nrZ6jH4fLg8VoQE0BoNFoNF6s8SHpnKAye+J2fEisbKFs2DUANDbWhu4X1erREf9R\ng8Pt0R27Go2mPWFZPaAmXp/VtoRimcW0sy4FoLmxAemdsAWIbvW4tMd/1OB0e/R8uxqNpj3WBLA3\nhCwaFVPJNEMRm/udR1pKCgBGVzM1zUFi3qnVo4X/iON0SR3xazSa9visnqBofqp7LQAnzLoGDEbc\nxljisLOrKkjk/QO4olk92uM/4jjcHswm7e9rNJowrAkg3SGWzfhUJxLB0CEj1AKLjXha2VUZZAl1\npVZPsDV0lNN3hV9H/BqNJhx/obaAqJvaahGxKWAwAmCISSDeYGdXZVDE35nVA8fULFx9Uh2dLt25\nq9FoIuAv1BZI6aSlGuLS/C+FJZ50i5OdlV2xeoKF/9ixe/qkOjrcHj1qV6PRtCea8NvSA68tNlLN\nTnZVRbB6XK3g8QSWO1tV4Tc4pjp4+6Q6OrXVo9FoIhHB6qGlJiTix2Ij0eBgb3ULLm8VABzNQQIf\nFOU7WyAmST0/mJROZ2voRaiX6JI6CiFmCyG2CiF2CCF+HWF9ihBikRCiUAixQggx2rt8mBBiXdBf\ngxDi7p5+E+E4XVKXa9BoNO2xhs67C0BLFcSlBl5bbNhEGy6PpKS2FTxuZeP4Lg7+Dl0XeJxBwn8Q\nVs87P4X5c7u/3yHSqfALIYzAY8AcYCRwmRBiZNhm9wPrpJRjgauBRwGklFullOOllOOBSUALsKgH\n2x8Ru9uDxWQ83KfRaDTHGv7pF70Rv5TtPH4s8cRINTBrV2VTwOaxZajHYNsHAsLfXaunrQGKPoTG\nfd18E4dOVyL+qcAOKeUuKaUDeB04P2ybkcAXAFLKIiBfCNEvbJtZwE4p5Z5DbHOnqM5dHfFrNJow\nwqdftDeAx9XO6jG7VSfursrmIOH39gM4w0bxxiSrx+5G/NuWqH2OQDZQV4Q/BygJel3qXRbMeuAi\nACHEVCAPyA3bZh7wWrSTCCFuEkKsEkKsqqysjLZZl9Aev0ajiUh4525LtXqMC+3cNTibSbVZVAev\nT+h9Eb9f+L2PsT7h72bEv/kd7369nw3UU+r4MJAshFgH3AGsBdy+lUIIC3Ae8Ga0A0gpn5JSTpZS\nTs7IyDikxuisHo1GExGLDRABq6fFW345zOrB7WBomkWldLazenzC763TczBWj70Rtn/a/f16iK6o\nYxkwIOh1rneZHyllg5TyJ14v/2ogA9gVtMkcYI2U8sAhtrdLOF064tdoNBEQQgl7u4g/1OoBGJZq\n6MTq8T52oXP31eV7+Xjj/sACn82TO/WojfhXAkOEEAXeyH0e8F7wBkKIZO86gBuAr6WUwZWQLqMD\nm6encbilFn6NRhMZa0LA42+uUo/BWT3ezJ8hKYKqJjvNzV4pa2f1hHv80b36Rz/fxuNfBs3ju/kd\niM+CgpNVLf9eLvdg6mwDKaVLCHE7sAQwAs9JKTcJIW7xrn8CGAG8KISQwCbget/+QggbcCZw82Fo\nf0Scbg9WbfVoNJpIBNfk7yDiL/AmAO2vrGYwRLB6wrN6IkfuLQ4XBxrs1LY4Vf+jq0XZPBOvBlMs\nIFUHs9HcI2+vK3Qq/ABSysXA4rBlTwQ9XwYMjbJvM5AWad3hwuHy6Dx+jUYTmXCrx2gJdPr61gNj\nMkwYBGwr2R8q/L6I35vOWSvjSIGolk1xldre4fKw7UAjo2o+V3X8R14AZau8x7L3qvD3ybBYZ/Vo\nNJqoWIMmY/Hl8AfP1ueN+BMNdqbkp7KjtEItj5LO+asPvBnqUTppi6sDNX82ltXDlvfBlgkDp4PR\nqlb08kQufU4dPR6JyyN1Vo9Go4lM8Ly74eUawC/8OJo5a1QWDY316rVvO0do526VKw4AVxSrZ7e3\nrn+cxUhhaT3U7YWs0aoaqMnbNaqF/9BweGtr6Ihfo9FExJoQ6vEHd+xC0CCvZs4a2Q+baAssN8UG\nSjN7I36XRXUGFJVURTzd7qpmMhOsjMtNVhG/vTFgLfki/l6erL3PqaPTK/y6LLNGo4mIJT6Q1dNS\n1UHE38SA1DhybRI7VhWhm2P9gl9Vp+4E5p40CoC1uyNnqxdXNZOfbmNMbhJb9jcig4XfpK2eHsHp\nVmlRunNXo9FExOrt3I1UpwdCrB6AE1IMNEkLVU12tc5r9Wz3ev9nT1R5LeXVDeyoCJ3IHZTHX5Bm\nY3ROEg6XB09bY6BmkK9DV0f8h4bD5Y34dZE2jUYTCWuCSp90tkBrXWi5BgBzqPDnxUtaZAxfbKkA\ncxw4m5FSUry/Cruwkpmi0jljDE7mLw8tRdbQ5qSqyUFBho2xOUkIPBicTe2tnl4exNXnhN/p9/h1\nxK/RaCJg8YpuXQkg20f8RhOYYvx2ULLZidMYwyebD/itnjV7a3G2tSjP32AAg4kRGTEsXF1Ki8Pl\nP1Sxt2M3P81GXlocmTFuBDLQj+Dr3O3lsg19Tvh9nbs6q0ej0UTEV5O/tlg9hnfugrcfQIm2cLRg\niU3gm+2VuEyx4Ghh0doy4g0OzLHeuwOjlTFZMTS2ufhgfaDMsi+jpyDdhhCCSf28Q6d0xN+z+K0e\n3bmr0Wgi4RPdOq8tEx7xg9fL92bvOJqJT0zC7vLw7Z5Wikr28/aaMvKTDBjMKpUTk4Usm4H8tDg+\n3BAQft/grbw0td24TKVLLrMv4vdl9eiI/5Bw6nROjUbTEZbwiD+S8AcifpzNJCUm8cSVk8hOTyXR\n6CQ/zcagZIOyfgCMFoTbwawR/Vi2s5pmu7J7iqubyU6KIcas+hxHpipdKm0x+fcDdMR/qPiFX1s9\nGo0mEr6MGp/w29Lbb2OxBUb3OloQFhuzR2cxdEA/suMki+86mWSTK0j4reB2MGtEJg63h2+2q5z+\n3d5UTh9DklXW4U7vmDCdztlDOFzqg9VWj0ajiYjf4/daPbGRPP5Qq8ef4mmOCy3Z4BN+kwXcDqbk\np5IYY+LzLSqnf3dVMwVBwt/PqgR+c423GqdRd+72CIHOXZ3Vo9FoIhBs9VjiwRwTYZsg4Xe2BFI8\nzbFhwu/1+I1WcNkxGw2cNiyTL4oqqG6yU9/qDBF+4R0xXFjp8e6nrZ4ewenSHr9Go+kAX+euszly\nRg94Pf4mNcjL0QwWr8BbbEr4PR7vBcFn9Zj9ds2sEZlUNztYvHILVhzkpwWE31cjaH2FRyWi6M7d\nnsGp0zk1Gk1H+CJ+iNyxC4GI32UH6Q61ekCVVXa1eevpowTcO/r2tKGZGA2CGV9fzc9Nb4Z4/L6x\nAbVuK9sONOqIv6fQRdo0Gk2HGAwB6yZ81K4Pn/D7bB1zmPA7W8Iifot/Bq6kODNT8lPI9FSSJyoY\nmBoXOK69EWm04sSkKnWadJG2HkHn8Ws0mk7x2T1RI/54FdG3edNv/FaP99HRHNq5a7SERO1nDM8k\njjYyza2h7oO3QFtSrJkNZXVBEb+2eg4JX5E2bfVoNJqo+DJ7ogm/b31zpXoMt3oczerC4B/AZQ3x\n6WcNS8EkPKQbW0KPa29EWBMYm5ukIn4hlPjriP/QcLjcgLZ6NBpNB/gj/midu16hb/KWWg63elpr\nvK+9GUFGS0jUXuA9fJohrFqnXRVoG5OTxNb9jbQ53d4xANEnaj8c9Dl11GWZNRpNp1g6ifh965u8\n0y6GWz3N3klX/OmcoVaPLxXU5m4MPa5dlWQem5uEyyMp2t/oHQOgI/5DQnfuajSaTvFF/JFG7UIg\n4o9m9bRUe18HDeAKTsn0jQFwtfonbgHA3gDWeMbkJgOwobROWz09ge7c1Wg0ndJp524nVk9LTehr\nozU0ancGJlintS7w3Nu5m50UQ6rNonz+MJuoN+iSOgohZgshtgohdgghfh1hfYoQYpEQolAIsUII\nMTpoXbIQYqEQokgIsUUIMaMn30A4TrcHk0FgMGirR6PRRKFTq8cn/FGsnhav1WPyevymMJ/eEdSp\n21obeO4VfiEEY3KS2FBWHzIGoLfoVPiFEEbgMWAOMBK4TAgxMmyz+4F1UsqxwNXAo0HrHgU+llIO\nB8YBW3qi4dFwuj3a5tFoNB3TWVaPJTyrx/s6mtVjNIeKtyM44g8Sfkdg9q2xuUlsr2jCYzg6I/6p\nwA4p5S4ppQN4HTg/bJuRwBcAUsoiIF8I0U8IkQScAjzrXeeQUtZxGHG6pU7l1Gg0HZMxHBKyITYl\n8rBU108AABA5SURBVPp2Vk9c6GO7zl2v1SO9xddCrB6vLeRyqBRQ7wxgY3KScHskrR7j0RfxAzlA\nSdDrUu+yYNYDFwEIIaYCeUAuUABUAs8LIdYKIZ4RQtiIgBDiJiHEKiHEqsrKym6+jQB2l474NRpN\nJ4y/HH6+GQxR5ub2C38lCENghG1Hnbug5vKFyBG/r8yzP+JXHbyNLuNRGfF3hYeBZCHEOuAOYC3g\nBkzAROBxKeUEoBlo10cAIKV8Sko5WUo5OSMj46Ab4nR7sOhUTo1G0xmiA53wWTvOZvXct63BoHx9\nf+du0MhdCETukTx+e4N69Ap/v0QrGQlW6p2i14Xf1IVtyoABQa9zvcv8SCkbgJ8ACCEEsBvYBcQB\npVLK5d5NFxJF+HsKp9ujrR6NRnNoGM0B+8YcF7rOHBfo3A2eiAUCAu6L7oUhSPi9Of1e4fd18NaW\niqPS6lkJDBFCFAghLMA84L3gDbyZO95LHjcAX0spG6SU+4ESIcQw77pZwOYeantEHNrq0Wg0PYHP\n7rHEtV/ualPPg+bcBQIC7mwBYVSdx37hD7V6AIZnJVDvEHh6Wfg7jfillC4hxO3AEsAIPCel3CSE\nuMW7/glgBPCiEEICm4Drgw5xBzDfe2HYhffO4HChs3o0Gk2PYIlXHbOWsG5JX5Qf/LxdxO+1iGJT\no0b8AMOyErBLE05HG9bD8Bai0RWrBynlYmBx2LIngp4vA4ZG2XcdMPkQ2tgtHDqrR6PR9AT+0brh\nwh90B2AK8/hDhD9O1QLy9QeEefwAw7MS2YAZt6OthxvfMX1OIR0utx61q9FoDp2OrB5QUb7BqzXh\nVo9vnt7YlMDI3QgRf0G6DRfmXrd6+pxCOt0Ss55vV6PRHCp+4Y9i9QRbPuFWj7NF3RnEpkRN5wRV\nPj4mNhZxjKZzHjWodM4+97Y0Gk1v4x+tG8XqCRF+s3ps5/GntPf4w46XYLNh9GjhPyR0Vo9Go+kR\nOrN6goU/fApFn8cfm6zGArjsSvgtCQF7yEtSQjxm6aSxrfdq8vc5hXS4PZh1565GozlUwksx+/Bb\nPUHLI1k9Po8flM9vbwixeXykJMRjFJJt+w5rNZsQ+pxCaqtHo9H0CP6IPz50eSSrxxQhq8dsU+mc\noNJCvZU5w0lLUsffvq+6p1reKX1OIZ0uqYVfo9EcOj6Rjmb1+EoyQ4SSDT6rxxfx1/qnXQwnKUEd\nb/e+mp5qeaf0OYVUVo/O6tFoNIdIt6yeSHn8tjDhbwyUgw5CePsHdh+obbfucNHnhN+pO3c1Gk1P\nENXq6aBz1+0At8tb4yeS8LeP+H39AyWVdUhfWefDTJ9TSIcu0qbRaHoCn+C3s3rCavNDqNXjq8Uf\nMeJPbH8e70Wjra2VysbeGcjVpxRSSqmEX0f8Go3mUIlassFn9UTw+N2OQElmS5yK8A2mTiJ+ta8F\nJ0X7G3vwDUSnTymk2yOREm31aDSaQyc5T5VVTh4Yujx84nUIzeP3TcLiq+Mfm6ImbnFEEX7vvhZc\nbO0l4e9SkbZjBadb+WPa6tFoNIdM1mi4dw/EhNkzlkgjd30RvzNg9fguDLEp0FAO0tO+vyBo38w4\nwdYDOuLvNg6XB9ARv0aj6SHCRR8i5/EbjKr+vjs44vfeGcSmQN1e9bwDq2dwqrnXIv4+pZAOtxJ+\nPfWiRqM5bPgnX4kNXW6yeq0en8cfSfijd+7mJ5tosrt6JbOnj1k9XuHXVo9GozlcWKNk+xgtoVZP\nsPA7vReDDiL+Syf24/IRp/V8eyPQpxRSWz0ajeawkzQAzv0HjDw/dLnREmr1BHv8PiIM4PJF/MZe\nLM3cJyN+LfwajeawIQRMjjCDrMkKLkcEjz81sE0HET+9KPx9SiEdWvg1Gs2RIjzi9wt/cmCbDtI5\n6cVZuPqUQvrSOa3a49doNL2Nyaqidp+f7+v8DbF6InTuhpd07gX6lEJqj1+j0RwxjOaA1WO2BSZc\nCRH+SBF/WGXPXqBPKWTA49fpnBqNppcxWgNWT3DGj0/4DeaArROy31Hq8QshZgshtgohdgghfh1h\nfYoQYpEQolAIsUIIMTpoXbEQYoMQYp0QYlVPNj4ch07n1Gg0RwqTVaVz+koy+/AJf6RoH45Oq0cI\nYQQeA+YAI4HLxP9v7+5i5KrLOI5/fzv7AlSxRTcNdBvaaFHWBopuGlRCRIjyFmrgwhIJhEAICSAa\nE8LLlVdyYQxcoA1BtBGliSjakAYUKukNAYotpKWt1FZpK9g1RqsQ3bfHi/Pf5eywW2Z2d+ZMz/l9\nkk3nvO38n+z22Wee85//SIN1p90L7IyIc4DrgQfrjl8UEWsiYmgBxjwrt3rMrDC1nrQ657vTF3Y7\nJc3qmS3xd3VlC7l1WKtnLbA/Ig5ExAiwCaibwMogsBUgIvYCKyQtXdCRNsBv4DKzwky1ev4zveLv\nOzVbzmGmG7vTru2gih9YBhzKbR9O+/JeBa4GkLQWOBMYSMcCeFbSK5Jume1JJN0iabuk7cPDw42O\nf5qpxO+K38zarbs33dx9d3qPX8qmdM5W8U9d21kVfyPuBxZL2gncAewAxtOxCyJiDVmr6DZJF870\nDSLi4YgYioih/v7+OQ1iqtXjit/M2m2yah955/2rcJ68ZOZ37U67tn2Jv5F37h4Blue2B9K+KRFx\nDLgRQJKAg8CBdOxI+veopCfJWkfb5j3yGYykefye1WNmbVfrTe2aeP/n9F507/R38NabfLXQJo2U\nxi8DqyStlNQLrAc250+QtDgdA7gZ2BYRxyQtkvThdM4i4MvAroUb/nSjqeLvq9Va9RRmZjObbNfU\nt3oAVl8DH79o9ms7reKPiDFJtwPPADXg0YjYLenWdHwDcDawUVIAu4Gb0uVLgSezFwF0Az+PiKcX\nPozM1JIN3a74zazNamk658TYzB+4ctxr08qebdLQIm0RsQXYUrdvQ+7xC8BZM1x3ADh3nmNs2Kin\nc5pZUWo9MPbfrN1T3+r5IG2+uVu61Tkl6O5yxW9mbdada9f0Ljr+ufXa3OopVWk8Mh701LpIrSUz\ns/ap5ZZjaDbxd+DN3RPGyNiE5/CbWTEmF1sDV/ztNDo+4amcZlaMWi7xN93j73PFP1ej4xNersHM\nipFP/HOa1eOKf05GxiY8o8fMipFfcrl+Hn8j17rin5uRcff4zawg82n11Ho6bpG2E4ZbPWZWmHm1\nenxzd87c6jGzwrjVU4zR8fCsHjMrRq3nvcdNT+f0zd05G3Grx8yKkn8DV0+zb+Dqy9b4mZhY2DHN\nolRZ0q0eMyvMZKun1ge1JlfDmfrA9fZU/aXKkqOe1WNmRZlM3s329+G9PxptWqitVFnSs3rMrDBT\nib/JGT35a9s0pbNUWXI0LdJmZtZ2k1V7s3P489e64m+ee/xmVpipir/JG7v5a13xNy+b1ePpnGZW\nACf+YvjmrpkVZnJZZrd62sutHjMrzOQ8/jlV/OlaV/zNW7fmDM5dvrjoYZhZFU1W7XOazpleLbSp\n4i/VZ+5+9+pzih6CmVVVVw3UNcfpnJMVv1s9ZmYnlk9cAsvXNn/dVMXfQa0eSZdK2idpv6S7Zzi+\nRNKTkl6T9JKk1XXHa5J2SHpqoQZuZtZxvv4LWH1N89d1WsUvqQY8BFwGDALXShqsO+1eYGdEnANc\nDzxYd/xOYM/8h2tmVkJT0zlH2/J0jVT8a4H9EXEgIkaATcC6unMGga0AEbEXWCFpKYCkAeAK4JEF\nG7WZWZm0+eZuI4l/GXAot3047ct7FbgaQNJa4ExgIB17ALgLOO56o5JukbRd0vbh4eEGhmVmVhKd\n1upp0P3AYkk7gTuAHcC4pCuBoxHxygd9g4h4OCKGImKov79/gYZlZnYCaPPN3Uamcx4Blue2B9K+\nKRFxDLgRQJKAg8AB4GvAVZIuB04CTpX0WERctwBjNzMrhw6s+F8GVklaKakXWA9szp8gaXE6BnAz\nsC0ijkXEPRExEBEr0nVbnfTNzOpMLdnQIRV/RIxJuh14BqgBj0bEbkm3puMbgLOBjZIC2A3c1MIx\nm5mVS1cNVGtbxd/QO3cjYguwpW7fhtzjF4CzPuB7PA883/QIzcyqoLuvo2b1mJlZq9V6Omoev5mZ\ntVqtr6Nu7pqZWat193XWWj1mZtZitV5X/GZmleKbu2ZmFVPr9SdwmZlViit+M7OKccVvZlYxtV5X\n/GZmldLd5zdwmZlViqdzmplVjG/umplVTBtv7ja0OqeZmbXYss+C2lOLO/GbmXWCoRtJH2TYcm71\nmJlVjBO/mVnFOPGbmVWME7+ZWcU48ZuZVYwTv5lZxTjxm5lVjBO/mVnFKCKKHsP7SBoG/jLHyz8G\n/H0Bh3MiqGLMUM24qxgzVDPuZmM+MyL6GzmxIxP/fEjaHhFDRY+jnaoYM1Qz7irGDNWMu5Uxu9Vj\nZlYxTvxmZhVTxsT/cNEDKEAVY4Zqxl3FmKGacbcs5tL1+M3M7PjKWPGbmdlxlCbxS7pU0j5J+yXd\nXfR4WkXSckm/l/S6pN2S7kz7T5P0O0lvpH+XFD3WhSapJmmHpKfSdhViXizpCUl7Je2R9Lmyxy3p\nW+l3e5ekxyWdVMaYJT0q6aikXbl9s8Yp6Z6U3/ZJ+sp8nrsUiV9SDXgIuAwYBK6VNFjsqFpmDPh2\nRAwC5wO3pVjvBp6LiFXAc2m7bO4E9uS2qxDzg8DTEfEp4Fyy+Esbt6RlwDeAoYhYDdSA9ZQz5p8A\nl9btmzHO9H98PfDpdM0PUt6bk1IkfmAtsD8iDkTECLAJWFfwmFoiIt6KiD+kx/8mSwTLyOLdmE7b\nCHy1mBG2hqQB4Argkdzussf8EeBC4EcAETESEf+k5HGTfTLgyZK6gVOAv1LCmCNiG/CPut2zxbkO\n2BQR/4uIg8B+srw3J2VJ/MuAQ7ntw2lfqUlaAZwHvAgsjYi30qG3gaUFDatVHgDuAiZy+8oe80pg\nGPhxanE9ImkRJY47Io4A3wPeBN4C/hURv6XEMdeZLc4FzXFlSfyVI+lDwC+Bb0bEsfyxyKZqlWa6\nlqQrgaMR8cps55Qt5qQb+Azww4g4D3iHuhZH2eJOPe11ZH/0zgAWSbouf07ZYp5NK+MsS+I/AizP\nbQ+kfaUkqYcs6f8sIn6Vdv9N0unp+OnA0aLG1wJfAK6S9GeyNt6XJD1GuWOGrKo7HBEvpu0nyP4Q\nlDnuS4CDETEcEaPAr4DPU+6Y82aLc0FzXFkS/8vAKkkrJfWS3QTZXPCYWkKSyHq+eyLi+7lDm4Eb\n0uMbgN+0e2ytEhH3RMRARKwg+9lujYjrKHHMABHxNnBI0ifTrouB1yl33G8C50s6Jf2uX0x2H6vM\nMefNFudmYL2kPkkrgVXAS3N+logoxRdwOfBH4E/AfUWPp4VxXkD28u81YGf6uhz4KNksgDeAZ4HT\nih5ri+L/IvBUelz6mIE1wPb08/41sKTscQPfAfYCu4CfAn1ljBl4nOw+xijZq7ubjhcncF/Kb/uA\ny+bz3H7nrplZxZSl1WNmZg1y4jczqxgnfjOzinHiNzOrGCd+M7OKceI3M6sYJ34zs4px4jczq5j/\nA6wUMGlvVZuTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2521c063cc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training is rather unstable; accuracy falling down after around 80 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 9664/10000 [===========================>..] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.9368\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 2. Deep CNN - 2\n",
    "- Measures used to ameliorate the performance of MLPs are employed\n",
    "    - Weight Initialization\n",
    "    - Batch Normalization\n",
    "    - Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.layers import BatchNormalization, Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def deep_cnn_advanced():\n",
    "    model = Sequential()\n",
    "    \n",
    "    model.add(Conv2D(input_shape = (X_train.shape[1], X_train.shape[2], X_train.shape[3]), filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same', kernel_initializer='he_normal'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same', kernel_initializer='he_normal'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size = (2,2)))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same', kernel_initializer='he_normal'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same', kernel_initializer='he_normal'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size = (2,2)))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same', kernel_initializer='he_normal'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same', kernel_initializer='he_normal'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size = (2,2)))\n",
    "    \n",
    "    # prior layer should be flattend to be connected to dense layers\n",
    "    model.add(Flatten())\n",
    "    # dense layer with 50 neurons\n",
    "    model.add(Dense(50, activation = 'relu', kernel_initializer='he_normal'))\n",
    "    model.add(Dropout(0.5))\n",
    "    # final layer with 10 neurons to classify the instances\n",
    "    model.add(Dense(10, activation = 'softmax', kernel_initializer='he_normal'))\n",
    "    \n",
    "    adam = optimizers.Adam(lr = 0.001)\n",
    "    model.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['accuracy'])\n",
    "    \n",
    "    return model    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = deep_cnn_advanced()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_34 (Conv2D)           (None, 28, 28, 50)        500       \n",
      "_________________________________________________________________\n",
      "batch_normalization_7 (Batch (None, 28, 28, 50)        200       \n",
      "_________________________________________________________________\n",
      "activation_4145 (Activation) (None, 28, 28, 50)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_35 (Conv2D)           (None, 28, 28, 50)        22550     \n",
      "_________________________________________________________________\n",
      "batch_normalization_8 (Batch (None, 28, 28, 50)        200       \n",
      "_________________________________________________________________\n",
      "activation_4146 (Activation) (None, 28, 28, 50)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_19 (MaxPooling (None, 14, 14, 50)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_36 (Conv2D)           (None, 14, 14, 50)        22550     \n",
      "_________________________________________________________________\n",
      "batch_normalization_9 (Batch (None, 14, 14, 50)        200       \n",
      "_________________________________________________________________\n",
      "activation_4147 (Activation) (None, 14, 14, 50)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_37 (Conv2D)           (None, 14, 14, 50)        22550     \n",
      "_________________________________________________________________\n",
      "batch_normalization_10 (Batc (None, 14, 14, 50)        200       \n",
      "_________________________________________________________________\n",
      "activation_4148 (Activation) (None, 14, 14, 50)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_20 (MaxPooling (None, 7, 7, 50)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_38 (Conv2D)           (None, 7, 7, 50)          22550     \n",
      "_________________________________________________________________\n",
      "batch_normalization_11 (Batc (None, 7, 7, 50)          200       \n",
      "_________________________________________________________________\n",
      "activation_4149 (Activation) (None, 7, 7, 50)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_39 (Conv2D)           (None, 7, 7, 50)          22550     \n",
      "_________________________________________________________________\n",
      "batch_normalization_12 (Batc (None, 7, 7, 50)          200       \n",
      "_________________________________________________________________\n",
      "activation_4150 (Activation) (None, 7, 7, 50)          0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_21 (MaxPooling (None, 3, 3, 50)          0         \n",
      "_________________________________________________________________\n",
      "flatten_11 (Flatten)         (None, 450)               0         \n",
      "_________________________________________________________________\n",
      "dense_21 (Dense)             (None, 50)                22550     \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 50)                0         \n",
      "_________________________________________________________________\n",
      "dense_22 (Dense)             (None, 10)                510       \n",
      "=================================================================\n",
      "Total params: 137,510\n",
      "Trainable params: 136,910\n",
      "Non-trainable params: 600\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 38min 26s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "history = model.fit(X_train, y_train, batch_size = 50, validation_split = 0.2, epochs = 100, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8nOWZ7//PNU1dVrVcZCwDxhU3hIHQYwI2ECCEmsMm\nkBCn0JKTPRtCzoY9u8lr+WWzWSCQeA0hySaAQwgkQEyvIQHjgnFv2MaWiyQX9TLt+v1xj+WxrJHG\ntozMM9f79fLLmqfM3Pdo9H3u53rKiKpijDEmc/gGugHGGGM+Xhb8xhiTYSz4jTEmw1jwG2NMhrHg\nN8aYDGPBb4wxGcaC3xhjMowFvzHGZBgLfmOMyTCBgW5AT8rKyrSqqmqgm2GMMZ8Yixcv3qWq5eks\ne0wGf1VVFYsWLRroZhhjzCeGiHyU7rJW6jHGmAxjwW+MMRnGgt8YYzJMnzV+EXkEuBSoU9WJPcwX\n4D7gYqANuFFVlyTmzUzM8wMPq+o9h9vQSCRCTU0NHR0dh/sUJkl2djaVlZUEg8GBboox5mOWzsHd\nXwMPAP+TYv4sYHTi32nAL4DTRMQPPAh8BqgBForIM6q66nAaWlNTQ0FBAVVVVbhtjTlcqsru3bup\nqalh1KhRA90cY8zHrM9Sj6q+BezpZZHLgf9R512gSESGAtOBDaq6UVXDwLzEsoelo6OD0tJSC/1+\nICKUlpba3pMxGao/avzDga1Jj2sS01JNP2wW+v3H3ktjMtcxcx6/iMwGZgMcd9xxA9waY4zpX9FY\nnDU7m3l/awN5IT8XTRhCXtbARHB/vOo2YETS48rEtGCK6T1S1bnAXIDq6upj7ouAGxoaeOyxx/jm\nN795SOtdfPHFPPbYYxQVFaVc5gc/+AHnnHMOF1xwwZE205gurZ1RXlldS0VhNuOHFVKYvf9Avqr2\nudfX2BahqSOCzyf4BAqyg+R3CypVRRV8vvT2IOuaOvjbh7sI+f3kZwcoyA4wuCCL8oIssgJ+OqMx\ntjd0UNvUQXFuiBElOeSGAqyrbea5D7bz0qpaSvJCzBhXwYyxgynKDVKzt51tDe10RGL4fULA56Ms\nP8S4oYVdwbqxvoXX1tSxvraFYEAI+f3kZfkZXJhNRUEWxXkhwtE47eEYbZEYDW1h9rZGaE703594\nD/a0RtjV0klDW5iJwwfxmfEVTK8qIeB3xZOOSIz1tS0s39bIyu2N1Dd30twRpbkzwsb6VtrCsa73\nIi+0gksnDeOC8RUMHZRNRWE2pXmhtN/LIyHpfNm6iFQBz6U4q+cS4FbcWT2nAfer6nQRCQDrgBm4\nwF8IfEFVV/b1etXV1dr9yt3Vq1czbty4Ptt6tGzevJlLL72UFStWHDA9Go0SCBwzO06HZKDf00wX\nicVRhVDg4IprLK5s2dPGutpmmtojVBbnMqIkh4DPx8LNe3hv0x42725lZGkuJ5bnc8LgfE4cnM+Q\nwmzCsTiPL9jCz17bwO7WcNdzVhbnoAqN7RFaOqOMKsujemQxp4wsJivoY29rhL1tYTbUtbBieyNb\n97Qf1K6CrABDBmWTFfSxpyXM7tYwkVicQTlBinNDlOSFKE8EeXl+FmWJ/zuiMZ5eso031tUTi/ec\nOQXZAZo7oimn+wSqq0rY2xpmfV1Ln++vCJxQnk8srmza1QpAWX4WsXjchXwkRoqmdMkN+VGFaDxO\nLK6U5IUoy88iLyvA8m2NhKNxCrID5AT9NLZH6IzGu9YtzA4wrCiHguwA+VkBjivJZdrIYqYdV0xt\nUwdPLNrKc8t2HLAxKM0LsfifP9Nn33ruryxW1ep0lk3ndM7HgfOAMhGpAe7GjeZR1TnAfFzob8Cd\nznlTYl5URG4FXsSdzvlIOqF/rLrzzjv58MMPmTJlCsFgkOzsbIqLi1mzZg3r1q3jiiuuYOvWrXR0\ndHDHHXcwe/ZsYP/tJ1paWpg1axZnnXUWf//73xk+fDh//vOfycnJ4cYbb+TSSy/lqquuoqqqii99\n6Us8++yzRCIR/vCHPzB27Fjq6+v5whe+wPbt2znjjDN4+eWXWbx4MWVlZQP8zph9Gtsj+H1CbtCP\nzyfE4sru1k7qmztd2HZEaeqIsq62mSUf7WX5tkY6o3HK8rMYVpRNdtBPc0eUls4IdU2dB4RId7kh\nP6PK8li6teGAsMwL+ckO+tndGub040v42YzRhKNxVm5vYs3OZoJ+YVBOkLxQgLW1zbyyupY/LK7p\nWl8ERpbkMqmyiOunH0d5fhaqEFOlqT3CjsYOtje0E4nFGVNRSGl+iKBfaGiL0NAeYU+L23C8s3E3\nDW2RA9o8uCCL2eccz6WThhLw+WjpjNDYHqG+uZPapk52t3RSkpfF8OIchhRms7ctzNa9bWzb287Y\nIQVcNHEIgwuyAdiyu4031tURjsYZXpTD8OIc8rMCRONKJBZnR0MHK7Y3smJbI7G4ctOZVZw/ZjAj\nSnK72hOLK7tbOtnZ1EFDW4TsoJ/soI/ckJ9BOSGKcoME/akPg7Z2Rvnr+l28ua4eVWVQTpDCnCCj\nyvKYOGwQI0pyUu5VjSjJpbqqhLs/O4F1tc3UNnVS29RBJJb6d96f0hrxf9z6GvH/v2dXsmp7U7++\n5vhhhdz92Qkp5yeP+N944w0uueQSVqxY0XU65J49eygpKaG9vZ1TTz2VN998k9LS0gOC/8QTT2TR\nokVMmTKFa665hssuu4wbbrjhoOD/zne+w2233cbPf/5zlixZwsMPP8ytt97K8OHD+d73vscLL7zA\nrFmzqK+vP6Lg/6SO+ONxZUN9C5XFrgzQn3Y2dvDqmlpeW13H2tpmJlcWcfrxJUwZUUxHNMbe1jAN\n7RFaO6O0hWM0tUdYV9vMqh1N1DZ1Ai4880IB2sLRHkeUIb+PicMLmXZcMQXZQXY0trO9sYPOSIyC\n7CAF2QHK8kOMrijgpIoCinKCbGtoZ8ueNjoiMaYdV8yEYYUE/D5UlfqWTjbUtfBhfSsf1rVQ39LJ\ntdUjOHt0WZ/lHFXlo91txFUpzg1RmBPE30+lhs5ojN0tYeqbO4nE4kwZUdRVEjH9r19H/KZn06dP\nP+Ac+Pvvv5+nn34agK1bt7J+/XpKS0sPWGfUqFFMmTIFgFNOOYXNmzf3+NxXXnll1zJPPfUUAG+/\n/XbX88+cOZPi4uJ+7c+AikWh4SMoPSHlIo3tEVZua+Tl1bW8sGInOxo7qCzO4T+umswZJ7j3eW9r\nmMcXbqFmbzshv4+sgI/CnCAVhdlUFGYR9PvY0xruGuVt3dPO1r1t1DV10hqO0tYZI5wYcY0oyWHi\nsEEs/mgvf1m+I2W7gn7hhPJ8zjyhjJOGFOATaOmI0tbeSm5OHuUFWZTlZzEoN0hhokY+ZJAb3R+K\nqrI8zuxhuogwuCCbwQXZfOqEQx8EiAhVZXmHvF46sgJ+hhXlMKwo56g8vzl8n8jg721k/nHJy9v/\nx/LGG2/wyiuv8M4775Cbm8t5553X4znyWVlZXT/7fT7aI5GDlklezu/3E40eXPNMi8YhHkuaIODz\nu+HooWqpg0AWZA9Kf532vbDiKfjgcbf+RT+CcZ+lrrmDDbUtfFjfQsuubZzV9BdO2vYUWW074Prf\nw5iZgBvVv7Gujsff28rumnWc3foK5/uXkqvTOfnEL/O1c47nN+98xPUPvcuNn6rCJ8K8hVtoC8co\nzQuRG2tgcmwlf42Ws0qrDmjaBNnEJP9mlhd8hqElgzhtVAn52QFyQ26kfe5J5Zw4OB8RQdXV2ldt\nbyIvK0BxrisB5GcFyM3yE/L7Dh5V71wOv7oCpt8MZ/zgwHkr/gjbIjDusxA6gsBt2wPrXoSqs6Ao\n6RyKljrYvQEqp4O/lz/vljrIKQZ/tyu3d3wA29+Hpu3QtA1C+TDkZKiYCIPHuc9Bf4jHobXe/csr\ng7zB4OthbyAeh7V/AfG7dgyqhFgEdq2FnSvAF3ADhtITIbtw/3qqEO2EaDsEc1O3u6MJ9nwInS1w\n3On73494zP2ulv8BysfC8efCsGmwbTGsewE2/w3Kx8BJM2H0hZBX2vPzJ9vXpkib+1vyHdrGvz99\nIoN/IBQUFNDc3NzjvMbGRoqLi8nNzWXNmjW8++67vT9ZtBNaaqGt0/2chjPPPJMnnniC7373u7z0\n0kvs3bvXrdu2ByLtEAu7P4h4BOJRF/zd+QIQyIFQDuSU9P6C8RhseBUWPgzrX3J/ECd8GiZ8jnh2\nCbXrF9K0aQktOUMZfMk/M2JohVsv3EbslX/Ft/gRJNZJvGwsEX8uWb+/gdeC5/IvLVdwmm81l/gW\ncJZvOQGJ81bsZEZJlI7ff5eHxlYwsryAp5bUUF9fx0M5D3CafoAGhbbC45nS9Dj4amHyXK45eRJ/\n/f1PGbPo2+RKJzfmVlB8fBUFHdtdKPgVLaxgy/96mx0dfndwLhtGP3kXgYbNkPcmzPgPqDrV9XHh\nL2HZIlh1nAuSiolI9U2MLC1mZGkipGNR+Ohv0NEAkQ73Po+ZBTmJs7aaa+Gx66CzCd7+LxhzCVSe\n4uatfwWe/LL7+bn/DROugKFTXDhF2qG4CiZ8bn9IqcKHr0JLPUy+bv9Gu7MFfvs52LHUPa6YCMOm\nwLYlUJe4MH7oFLjsfhg6+cDf69b34K2fwPoXXR8/868w5mL3eXz5B7Ds92458UF+hQvGiDswij/L\nPd+I6VA0MtHuDigcCpOuTW+jsPYFeOn7sPcj91lN/mwOqoQJV8L02e45a1fBs3dAzXv7l8seBOG2\nA9fteo7g/vcoFgESdbasQXDKl+C0r0HBMPjobVj6eOK9rd2/fm4ZnHw1DJ0Ef7sf6lfDoBHw4evw\n9/v3LxfMc+/Blndh1Z8AcRvIYI77N/JTUP0VqKx2v9dl8+DdObBr3f42FY+CKx+CEafuf97mWjdo\nGH30z+6z4E9TaWkpZ555JhMnTiQnJ4eKioqueTMvvJA5P3+AcePGMmbMWE4//fTUTxSLuBGZqgvo\n+jUuwONRF+LxGDTWQJ7fhUzC3XffzfXXX89vf/tbzjh1KkMGl1HQXgPxECDgD7lwDuW5PyJfwI2S\n9g1GVSHa4UYbLfVuxNfSBOu3uUBPHm1tfJPos98msPdDwtllbD7pq3R2tHHc5pcZtO4FfMBQIK5l\njOZVauf8hf8ovpXCsiF8dtMPGRbbzu+j5/Hb2AWsqBlFgBjfDDzD7TzNW1lvAhAtPA7fxNvYceI1\naLSC9Uuf4NOr7kJXP8N/LKlmwrBCnjvxWUZsWwHnfx+ZfD15gyph8a/h+e/CL84gN9LBReFm2itP\nI15yIoPbd0LTFjeCPP/7MGg48qdvMHLNQ4z89Pdd3xb8NzRshnP+jxvN/c/lkFsKbbshfwiMvdSN\ndmsWuRHf3++H874HU29wy7/9X7B384G/0/whcPGP3chv3hegfQ/c+Bz88avw51vga2+65396Ngye\nADP/3T3Xyj/B0kcPfK6XfwDTv+pGwO/+3H0+ADa8DJc94H7HT3zRBcRlP4P2BjfyX/UsDJ8Gk65x\nI/nXfgRzz088Vxk0boO61bD1XbfRP/MOWPu8a2/lqVC3BmKdcPZ34JQboWCoe614HPZuSuwJLHEb\njvcecssme/PHcN6dMOk6t6Fo2u4+6xUT93+23p0DL34PBo+HM25xQZ9XBq273PJ1q+Bv98LffwbH\nnwcb33Cj+CvmQMnxULscaldCVqEb/Q852W14d2+AXevdxnYfX8CFcCDHbTjeecC9n3mDoXk7hArc\nBnvwOCgb7dZZ/gdY9Es3iCo9Ea76FYy/wm3gtrwD25e6DWrVWRDMdu/Nzg/cAKlttwv5jgZY/azb\n062Y6PrVvsetd84/ur0PfxAWzIVHLoJzvwtjL3bvzfInXJu/s849/1H0iTy4e9Q17XC7oDlF7o8k\nlNdziUTjLqybd+4fgYTyE+uVHrzrGo+6D+i+D5YvAA1bIJx0apokdv80UabJKoTikXRGYvj9fgLR\nVt556Wm+cdf/x9IFb7m2BXIOrYQTi0LbLlavWsG45z8PRcfRPu4aXpdqSlb8mtObXmBzvIKfRK/h\nxfipRBLjg9ygcF7BVkaXhBg96TQ+NeFEYlsXIs/eQVnregDq/BW8PPoHaNXZRGNxwrE4uaEAF4yr\nYEj7ejeyPv48t9uc3OZ4DH5xJqox6m54g8HbXkb+8CX3h3H+XQe2f+cKeOFONyI94xYXeKk8+WVY\nMx9uWwxZ+XD/VKiYAF98xm1w3/mZC7WTr3Yj3+TSx45l8NL/hU1vutFkPALDpsKZ33K/v2CO24A+\n/0+wcxkUVkJTDVz7O1fKWfciPHYNnPW/YesCFxyz34Dyk9zzRzrc7z6YA4Fs2PSWC6gNr7j5FSfD\np251JZdX/82NtkuOh5VPwWfvd6PYVNr3uo3IksQttnJLXdCefDWccpN7L2IRtyH9230ujGf+e6/H\nWbpEO92eQDDbffY2vwWv/qsrEflD7vO9T+FwF56RVvdaYy+FK+emLnPt2QQL5riN7okXwIU/Sq+M\n0pe9H8F7c91Ge/wVMPYSCOUevFzbHrcBGnF676Wy3nQ2w7InYOljUDAETv+m2wtI/rx3NMJf/tGF\nPbj3cer/csum8zvowaEc3LXg725fzS+Q40Y1Gnd/lKWjD/wgRDrccrGw2/XLH+xGBu0NbmQdynd/\npPvqeLEI7NnoRgUlx++vR6q60UI85v4Yg4kPY6TNtaWlFvwh1u9Rrrn+C8QjHYRCIX4+5yFOPa2X\nPYtuorE4PpGui0PiqqxcuYr4lr8RWj6PMW1L8IkSxc+rJddSN+V2KgeXUpgToDA7SHlBFoNygj2f\nJRKLwII5aHMtct53IavgcN55WPUMPPEP8Jl/cyProhFw86sH16EPxd6P4IFTYeLnXfngr//pwnfY\n1PTWV3Ubq9XPuDLECZ8+eCMbi7rR5Js/hnP/jxtN7/PU7P3lk8/9tyvZ9KV+rQuGylP3v9ba5+GP\nN7sNxXnfc6PrdLTudgEXPMoHWFVhzV/go7+7sCsc5j4Xq59xG7JYGM641ZWWBrC2fcxZ/Zwb/E2+\nDnL7KL/2wYI/XZ0tLmBzS92HMRZ2f3S+AJSNAdSNnBprXJiVHO/+EONxd3ApHoWi49yoPDkM2va4\nX2YwB0pOcK/R8JHbiBSN3F8PTreNezbuf37xQdlJfYZhNBZPnBMepbUz2nW2yr7wj8Xi7Nyyka8+\ns4PSvBD/MD7ANYNWMnTieciQg67T+3iown+f40bP/iz42lsweOyRP+/LP3A1W38Ixl8Gn3/4yJ+z\nJ/H4wXt5bXtg7rlu9Hrpfx3Z89evcwcXk+v9nwQdjW6vuHzMQLfE0yz407V7g9st8wXdaLBtjwvp\nsjEH1tha6tzu9qBKyCt3od6224V68pkEyToa3W6rL+BKBIFsd/DucEZekQ4X/vGoq0cGc4jG4jS2\nR7quFswJ+skN+Qn4hcb2KC0dURQl4BPysgJJVyAq8bgSCPjYtmk9FA1nUmVRrxeqfKzWvwKPfh4u\n+nc449Buj5FSe4Mr8YRb4NZFUDyyf543XbHo4ZcNjEmTncefrmjn/tJKwxb3f9HIgw+s5JW7DUTj\ntkR9fLerL6cKfXBnH5Se6A6M5Za6WmeKXVxV7QrkuGrXRT/77qfi9wXxl55ELBalqQOaG1po64yh\nKFkBP3lZAdrDMZo63HGGkN9HWUGIopwg2UF/yot49oYCjBt5ZLuX/W70BfCdta5c0F9yiuD6eW7v\n7eMOfbDQN8eczP1EqroaZE6xO4Oho8GNqHuqs4m4kk79GmjZ6Wr6BUP7fo2sfHdkv5fd8nA0zqZd\nrXRGYymX6S4n6O+quWcH959HHo3FicaVrEAP55Z/kvRn6O9z3Gn9/5zGfEJlbvDHwoC6WrKI2wD0\nxh90pZrmnW6vIN1gTVqurTNKwO/ruilXOBpjY30rMVWGDcrB7xd8Ioi4szD3XUAUi7s9Ap8IBVkB\ngj3c1Asg4PcRsONmxpg+HCOF3QGw78KpQCj9dbIKXI09jXXy8/MB2L59O1dddRUNbWE21LewdmcT\nH+1upaEtzFnnnMcHS5cwqiyPsoIsinND7kZP2cGuW+D+cs6DhIhSlp9FSV6Iyy+7lIaGhsPpsTHG\nAF4N/kg77NrgRuep7LsApb8uQU9h2LBhPPK7x9m6p528rABlBVm0dkbZsqcNUIYXZfd6o7F7772X\ntra2rsfz58/v9d7+xhjTF28FfzzurpSrX+vO4Gje4Q7E9iQaxt2/Jr1zxO+8804efPDBrsf/8i//\nwg9/+ENmzJjBtGnTOPnkk/nzn/980Hqr1m7glCmTyQ76qMgVvv21m/j8jNO565tfgmiY7KAL/W98\n4xtUV1czYcIE7r77bsDd+G379u2cf/75nH/++YC7zfOuXbsA+OlPf8rEiROZOHEi9957L+DuIjpu\n3Di++tWvMmHCBC688ELa2w++r7oxJnN9Mmv8z9/pLlk/gLpTMTXuwjwQgkinuwI2mLP/ith9ou1u\n2WDiCsIhJ8Ose1K+5LXXXsu3vvUtbrnlFgCeeOIJXnzxRW6//XYKCwvZtWsXp59+OudeMJP2qPuC\njXU7m9mytw3B3V3xZ/fdS25uLqtXr2bZsmVMm7b/itMf/ehHlJSUEIvFmDFjBsuWLeP222/npz/9\nKa+//vpBt19evHgxv/rVr1iwYAGqymmnnca5555LcXEx69ev5/HHH+ehhx7immuu4Y9//CM33HDD\n4b7bxhiP8dCIX5Luz5EN+NxpmSKu9NP9pmUadxdDpWnq1KnU1dWxfft2PvjgA4qLixkyZAh33XUX\nJ588ibPP+zQ1NdtYsnYzu1rcJeuhgI+S3BDBgI+g38dbb73VFcCTJk1i0qRJXc//xBNPMG3aNKZO\nncrKlStZtWpVr+15++23+dznPkdeXh75+flceeWV/PWvfwXSv/2zMSYzfTJH/L2MzA8S7XBXPIby\nofR4N03V3YclrwwGDU/7qa6++mqefPJJdu7cybXXXsujjz5KXV0dj81/Hb8/yMwzTmZInp+xwwoR\ncaN8WrLp6/yfTZs28ZOf/ISFCxdSXFzMjTfe2ONtndN1wO2f/X4r9RhjDuChEX8KgWx3AU+4xQU+\nJG6oFj+0M3pw5Z558+bx5JNPcvXVV9PQ0EB2YQn4Amxe/h5bt2whNyuAL8Wpnueccw6PPfYYACtW\nrGDZsmUANDU1kZeXx6BBg6itreX555/vWifV7aDPPvts/vSnP9HW1kZraytPP/00Z5999iH1xxiT\nmT6ZI/5DFcp3B3n3fSlDNHH3QP+hndEzYcIEmpubGT58OEOGDGHGZz/Pr377ea698ExOm34qY8f2\nfl+Zb3zjG9x0002MGzeOcePGccop7j7tkydPZurUqYwdO5YRI0Zw5pn7v2tp9uzZzJw5k2HDhvH6\n6693TZ82bRo33ngj06dPB+Dmm29m6tSpVtYxxvQprXv1iMhM4D7cl6Y/rKr3dJtfDDwCnAB0AF9W\n1RWJed8GbsZ9A8Fy4CZV7bWO0e/36ol2ulutFlZCfrnbCDRscbeiPYTTOTujMeqaOumIxOiMxomr\nUp6fxdBP6FfLfVK/c9cYc7BDuVdPn6UeEfEDDwKzgPHA9SIyvttidwFLVXUS8EXcRgIRGQ7cDlSr\n6kTchiON+9L2M3/Inemz7773+y7eOoTb/e67yraxPYLfJ5TkhRhRksuQQUf3CxOMMaa/pVPqmQ5s\nUNWNACIyD7gcSD7tZDxwD4CqrhGRKhHZ9xVVASBHRCJALrC9vxqfNhF335zO5sQ3UYXdxiDNs3rC\n0Tgb61uJq3JCeR45vVxwZYwxx7p0km84sDXpcU1iWrIPgCsBRGQ6MBKoVNVtwE+ALcAOoFFVX+rp\nRURktogsEpFF9fX1PTbkiG4hHcpzN2GLdbp/aZZ43E3UWojFlVFl3gn9Y/F23MaYj0d/ndVzD1Ak\nIkuB24D3gVii9n85MAoYBuSJSI9XEqnqXFWtVtXq8vLyg+ZnZ2eze/fuww+skLt3DuFWV+pJ48Bu\nS0eUDXUtRGJKVVler7dW+CRRVXbv3k12tpWpjMlE6STZNmBE0uPKxLQuqtoE3AQg7n7Am4CNwEXA\nJlWtT8x7CvgU8LtDbWhlZSU1NTWk2hvokyo07YbtiW/dyumArJaUizd3RGlqjxDwu3r+liZvnfma\nnZ1NZWXlQDfDGDMA0gn+hcBoERmFC/zrgC8kLyAiRUCbqoZxZ/C8papNIrIFOF1EcoF2YAZw4Ok6\naQoGg4waNepwVt3v8R/C+hddyee6x2Ds9IMWicbi3PnUcp5cXMOsiUP48VUnU5B9BN/5aowxx5g+\ng19VoyJyK/Ai7qycR1R1pYh8PTF/DjAO+I2IKLAS+Epi3gIReRJYAkRxJaC5R6Un6Rh5Bqz9i/u5\n5PiDZndGY3xr3lKeX7GT22eM5tsXjP5kf6GJMcb0IK2itarOB+Z3mzYn6ed3gJNSrHs3cPcRtLH/\nHPep/T8XVx0wqy0c5Wu/Xcxf1+/iny8dz1fOOsK9C2OMOUZ542hluoZOcnfjzC484EvPVZVvzVvK\n3zbs4sefn8Q1p47o5UmMMeaTLbOC3x+EE2ccdKfO3y3Ywkuravm/l4yz0DfGeF5mBT/AVY8c8HDt\nzmZ++Nwqzj2pnC+faeUdY4z3ZV7wJ92moSMS47bHl1CQHeQnV0/G57MDucYY78u84E9y7yvrWVfb\nwv98eTrlBUf3u3eNMeZY4a2rkg6BqvLM0m1cMK6Cc046+EphY4zxqowN/nW1LWxv7OCCcYMHuinG\nGPOxytjgf21NHQDnj7XgN8ZklowN/tfX1jF+aCEVhXajMmNMZsnI4G9sj7D4o72cP9Zq+8aYzJOR\nwf/X9fXE4sqnrcxjjMlAGRn8r6+ppyg3yJQRxQPdFGOM+dhlXPDH48qb6+o4Z3Q5frtgyxiTgTIu\n+Jdva2RXS9jKPMaYjJVxwf/amjpEsIu2jDEZK6OCf/WOJn7zzmZOHVlCSV5ooJtjjDEDImOCf0Nd\nMzc8vIDsgJ//vGbyQDfHGGMGTEYE/+ZdrXzhoQWICI999TRGlOQOdJOMMWbApBX8IjJTRNaKyAYR\nubOH+cXTE8GUAAASzklEQVQi8rSILBOR90RkYtK8IhF5UkTWiMhqETmjPzuQjn/8wwdEYnEevfk0\nji/P/7hf3hhjjil9Br+I+IEHgVnAeOB6ERnfbbG7gKWqOgn4InBf0rz7gBdUdSwwGVjdHw1PVzQW\nZ9m2Rq46pZIxQwo+zpc2xphjUjoj/unABlXdqKphYB5webdlxgOvAajqGqBKRCpEZBBwDvDLxLyw\nqjb0W+vTsHl3G+FonDFDCj/OlzXGmGNWOsE/HNia9LgmMS3ZB8CVACIyHRgJVAKjgHrgVyLyvog8\nLCJ5R9zqQ7B2ZzMAY220b4wxQP8d3L0HKBKRpcBtwPtADPcNX9OAX6jqVKAVOOgYAYCIzBaRRSKy\nqL6+vp+aBWt3NuETOHGw1faNMQbSC/5twIikx5WJaV1UtUlVb1LVKbgafzmwEbd3UKOqCxKLPonb\nEBxEVeeqarWqVpeX99/FVWt2NlNVlkd20N9vz2mMMZ9k6QT/QmC0iIwSkRBwHfBM8gKJM3f2XRF1\nM/BWYmOwE9gqImMS82YAq/qp7WlZs7PZyjzGGJOkzy9bV9WoiNwKvAj4gUdUdaWIfD0xfw4wDviN\niCiwEvhK0lPcBjya2DBsBG7q5z6k1NoZZcueNq46pfLjekljjDnm9Rn8AKo6H5jfbdqcpJ/fAU5K\nse5SoPoI2njY1tW6A7t2Gqcxxuzn6St37YweY4w5mKeDf83OZnJDfkYU2y0ajDFmH08H/9qdzYyu\nKMBnX7hijDFdPBv8qsra2mbGVliZxxhjknk2+OtbOtnTGrYDu8YY041ng98O7BpjTM88H/w24jfG\nmAN5NvjX7GymLD+L0vysgW6KMcYcUzwb/GvtVg3GGNMjzwZ/XXMHw4tyBroZxhhzzPFs8EdiSijg\n2e4ZY8xh82wyRqJxgn7Pds8YYw6bZ5MxHIsTDNgVu8YY051ngz8SixOyEb8xxhzEk8kYjcWJK1bq\nMcaYHngyGSMxBSz4jTGmJ55MxnAsDkDQbzV+Y4zpzpPBH0kEv53OaYwxB/NkMka6Rvye7J4xxhyR\ntJJRRGaKyFoR2SAid/Ywv1hEnhaRZSLynohM7DbfLyLvi8hz/dXw3kSiVuM3xphU+kxGEfEDDwKz\ngPHA9SIyvttidwFLVXUS8EXgvm7z7wBWH3lz02M1fmOMSS2dIfF0YIOqblTVMDAPuLzbMuOB1wBU\ndQ1QJSIVACJSCVwCPNxvre5DV43fRvzGGHOQdJJxOLA16XFNYlqyD4ArAURkOjASqEzMuxf4JyDe\n24uIyGwRWSQii+rr69NoVmp2cNcYY1Lrr2S8BygSkaXAbcD7QExELgXqVHVxX0+gqnNVtVpVq8vL\ny4+oMXZw1xhjUgukscw2YETS48rEtC6q2gTcBCAiAmwCNgLXApeJyMVANlAoIr9T1Rv6oe0phe3g\nrjHGpJROMi4ERovIKBEJAdcBzyQvICJFiXkANwNvqWqTqn5PVStVtSqx3mtHO/QhudRjB3eNMaa7\nPkf8qhoVkVuBFwE/8IiqrhSRryfmzwHGAb8REQVWAl85im3uk5V6jDEmtXRKPajqfGB+t2lzkn5+\nBzipj+d4A3jjkFt4GCz4jTEmNU8mY9hu0maMMSl5MhkjUTuP3xhjUvFkMnaVeuzgrjHGHMSTwR+2\nGr8xxqTkyWQMRy34jTEmFU8m475v4LIavzHGHMyTyRixu3MaY0xKng1+EfD7LPiNMaY7TwZ/OBYn\n6PfhbhtkjDEmmSeDPxJVsqy+b4wxPfJkOkZicYJ2L35jjOmRJ9MxEovbgV1jjEnBk8G/r8ZvjDHm\nYJ5Mx0hM7Rx+Y4xJwZPpGInaiN8YY1LxZDq6g7tW4zfGmJ54Mvitxm+MMal5Mh0jFvzGGJNSWuko\nIjNFZK2IbBCRO3uYXywiT4vIMhF5T0QmJqaPEJHXRWSViKwUkTv6uwM9sYO7xhiTWp/pKCJ+4EFg\nFjAeuF5Exndb7C5gqapOAr4I3JeYHgW+o6rjgdOBW3pYt9/ZefzGGJNaOsPi6cAGVd2oqmFgHnB5\nt2XGA68BqOoaoEpEKlR1h6ouSUxvBlYDw/ut9SmE7aweY4xJKZ10HA5sTXpcw8Hh/QFwJYCITAdG\nApXJC4hIFTAVWHB4TU1f2G7ZYIwxKfVXOt4DFInIUuA24H0gtm+miOQDfwS+papNPT2BiMwWkUUi\nsqi+vv6IGhOJxa3Gb4wxKQTSWGYbMCLpcWViWpdEmN8EIO5eyJuAjYnHQVzoP6qqT6V6EVWdC8wF\nqK6u1vS7cLBIVK3Gb4wxKaQzLF4IjBaRUSISAq4DnkleQESKEvMAbgbeUtWmxEbgl8BqVf1pfza8\nN5FYnJCVeowxpkd9jvhVNSoitwIvAn7gEVVdKSJfT8yfA4wDfiMiCqwEvpJY/UzgH4DliTIQwF2q\nOr+f+3EAu4DLGGNSS6fUQyKo53ebNifp53eAk3pY723gY6+5WI3fGGNS82Q6RmJqI35jjEnBc+kY\niyuxuAW/Mcak4rl0jMTiAHZ3TmOMScGzwW81fmOM6Znn0jESc5cAWKnHGGN65rl07Cr1WPAbY0yP\nPJeO4ei+4LcavzHG9MRzwd9V47crd40xpkeeS0er8RtjTO88l45W4zfGmN55Lh3DMavxG2NMbzwX\n/JGoncdvjDG98Vw6hu3grjHG9Mpz6Wg1fmOM6Z3n0jEctbN6jDGmN55Lx/3n8dvBXWOM6Ylng99G\n/MYY0zPPpaMFvzHG9M5z6Ri2K3eNMaZXaaWjiMwUkbUiskFE7uxhfrGIPC0iy0TkPRGZmO66/c3O\n4zfGmN71mY4i4gceBGYB44HrRWR8t8XuApaq6iTgi8B9h7Buv7Jv4DLGmN6lMyyeDmxQ1Y2qGgbm\nAZd3W2Y88BqAqq4BqkSkIs11+5XV+I0xpnfppONwYGvS45rEtGQfAFcCiMh0YCRQmea6JNabLSKL\nRGRRfX19eq3vwb4af8BnI35jjOlJfw2L7wGKRGQpcBvwPhA7lCdQ1bmqWq2q1eXl5YfdkEgsTsjv\nQ8SC3xhjehJIY5ltwIikx5WJaV1UtQm4CUBc4m4CNgI5fa3b3yLRuN2Z0xhjepHOiH8hMFpERolI\nCLgOeCZ5AREpSswDuBl4K7Ex6HPd/haJxQnaDdqMMSalPkf8qhoVkVuBFwE/8IiqrhSRryfmzwHG\nAb8REQVWAl/pbd2j0xUnHFM7sGuMMb1Ip9SDqs4H5nebNifp53eAk9Jd92jaV+M3xhjTM88lZCQW\nt3vxG2NMLzyXkGE7uGuMMb3yXPBHYnGr8RtjTC88l5B2cNcYY3rnuYSMRO3grjHG9MZzCenO47ca\nvzHGpOLN4LcRvzHGpOS5hLQavzHG9M5zCWkXcBljTO88l5Cu1GM1fmOMScV7wR+1Gr8xxvTGcwkZ\njqndndMYY3rhuYS0Gr8xxvTOcwlpNX5jjOmdJ4Pf7s5pjDGpeSohVZWIncdvjDG98lRCRmIKYMFv\njDG98FRCRmJxADu4a4wxvUgrIUVkpoisFZENInJnD/MHicizIvKBiKwUkZuS5n07MW2FiDwuItn9\n2YFk+4LfDu4aY0xqfQa/iPiBB4FZwHjgehEZ322xW4BVqjoZOA/4TxEJichw4HagWlUn4r5w/bp+\nbP8BwtFE8NvBXWOMSSmdhJwObFDVjaoaBuYBl3dbRoECEREgH9gDRBPzAkCOiASAXGB7v7S8B+Gu\nEb8FvzHGpJJOQg4HtiY9rklMS/YAMA4X6suBO1Q1rqrbgJ8AW4AdQKOqvnTErU5h38Fdq/EbY0xq\n/ZWQFwFLgWHAFOABESkUkWLc3sGoxLw8EbmhpycQkdkiskhEFtXX1x9WIyI24jfGmD6lk5DbgBFJ\njysT05LdBDylzgZgEzAWuADYpKr1qhoBngI+1dOLqOpcVa1W1ery8vJD7QeQVOO3g7vGGJNSOsG/\nEBgtIqNEJIQ7OPtMt2W2ADMARKQCGANsTEw/XURyE/X/GcDq/mp8d10jfju4a4wxKQX6WkBVoyJy\nK/Ai7qycR1R1pYh8PTF/DvBvwK9FZDkgwHdVdRewS0SeBJbgDva+D8w9Ol2xGr8xxqSjz+AHUNX5\nwPxu0+Yk/bwduDDFuncDdx9BG9NmNX5jjOmbpxIybBdwGWNMnzwV/JGojfiNMaYvnkrIfTX+LDu4\na4wxKXkqIa3Gb4wxffNUQobtdE5jjOmTpxLS7s5pjDF981bwR+1+/MYY0xdPJaR9A5cxxvTNUwlp\nt2U2xpi+eSohrcZvjDF981Twh6Nxgn7B3Q/OGGNMTzwV/JFY3Mo8xhjTB0+lZCSmFvzGGNMHT6Vk\n2Eb8xhjTJ0+lZCQaJ2QHdo0xplfeCv5Y3G7XYIwxffBUSlqN3xhj+uaplAzH4na7BmOM6YOnUtJK\nPcYY07e0UlJEZorIWhHZICJ39jB/kIg8KyIfiMhKEbkpaV6RiDwpImtEZLWInNGfHUgWidnBXWOM\n6UufwS8ifuBBYBYwHrheRMZ3W+wWYJWqTgbOA/5TREKJefcBL6jqWGAysLqf2n6QSNRq/MYY05d0\nUnI6sEFVN6pqGJgHXN5tGQUKxN0rIR/YA0RFZBBwDvBLAFUNq2pDv7W+GzuP3xhj+pZOSg4HtiY9\nrklMS/YAMA7YDiwH7lDVODAKqAd+JSLvi8jDIpLX04uIyGwRWSQii+rr6w+1H4DdssEYY9LRXyl5\nEbAUGAZMAR4QkUIgAEwDfqGqU4FW4KBjBACqOldVq1W1ury8/LAaEYnFCQWsxm+MMb1JJ/i3ASOS\nHlcmpiW7CXhKnQ3AJmAsbu+gRlUXJJZ7ErchOCrsPH5jjOlbOim5EBgtIqMSB2yvA57ptswWYAaA\niFQAY4CNqroT2CoiYxLLzQBW9UvLe+Buy2zBb4wxvQn0tYCqRkXkVuBFwA88oqorReTriflzgH8D\nfi0iywEBvququxJPcRvwaGKjsRG3d3BUWI3fGGP61mfwA6jqfGB+t2lzkn7eDlyYYt2lQPURtDFt\nYTuP3xhj+uSp4XHESj3GGNMnT6XkhROGMGF44UA3wxhjjmlplXo+Kf7r2ikD3QRjjDnmeWrEb4wx\npm8W/MYYk2Es+I0xJsNY8BtjTIax4DfGmAxjwW+MMRnGgt8YYzKMBb8xxmQYUdWBbsNBRKQe+Ogw\nVy8DdvW5lLdkYp8hM/udiX2GzOz3ofZ5pKqm9WUmx2TwHwkRWaSqH8tN4Y4VmdhnyMx+Z2KfITP7\nfTT7bKUeY4zJMBb8xhiTYbwY/HMHugEDIBP7DJnZ70zsM2Rmv49anz1X4zfGGNM7L474jTHG9MIz\nwS8iM0VkrYhsEJE7B7o9R4uIjBCR10VklYisFJE7EtNLRORlEVmf+L94oNva30TELyLvi8hziceZ\n0OciEXlSRNaIyGoROcPr/RaRbyc+2ytE5HERyfZin0XkERGpE5EVSdNS9lNEvpfIt7UictGRvLYn\ngl9E/MCDwCxgPHC9iIwf2FYdNVHgO6o6HjgduCXR1zuBV1V1NPBq4rHX3AGsTnqcCX2+D3hBVccC\nk3H992y/RWQ4cDtQraoTAT9wHd7s86+Bmd2m9djPxN/4dcCExDo/T+TeYfFE8APTgQ2qulFVw8A8\n4PIBbtNRoao7VHVJ4udmXBAMx/X3N4nFfgNcMTAtPDpEpBK4BHg4abLX+zwIOAf4JYCqhlW1AY/3\nG/fNgDkiEgByge14sM+q+hawp9vkVP28HJinqp2qugnYgMu9w+KV4B8ObE16XJOY5mkiUgVMBRYA\nFaq6IzFrJ1AxQM06Wu4F/gmIJ03zep9HAfXArxIlrodFJA8P91tVtwE/AbYAO4BGVX0JD/e5m1T9\n7NeM80rwZxwRyQf+CHxLVZuS56k7Vcszp2uJyKVAnaouTrWM1/qcEACmAb9Q1alAK91KHF7rd6Km\nfTluozcMyBORG5KX8VqfUzma/fRK8G8DRiQ9rkxM8yQRCeJC/1FVfSoxuVZEhibmDwXqBqp9R8GZ\nwGUishlXxvu0iPwOb/cZ3KiuRlUXJB4/idsQeLnfFwCbVLVeVSPAU8Cn8Hafk6XqZ79mnFeCfyEw\nWkRGiUgIdxDkmQFu01EhIoKr+a5W1Z8mzXoG+FLi5y8Bf/6423a0qOr3VLVSVatwv9vXVPUGPNxn\nAFXdCWwVkTGJSTOAVXi731uA00UkN/FZn4E7juXlPidL1c9ngOtEJEtERgGjgfcO+1VU1RP/gIuB\ndcCHwPcHuj1HsZ9n4Xb/lgFLE/8uBkpxZwGsB14BSga6rUep/+cBzyV+9nyfgSnAosTv+09Asdf7\nDfw/YA2wAvgtkOXFPgOP445jRHB7d1/prZ/A9xP5thaYdSSvbVfuGmNMhvFKqccYY0yaLPiNMSbD\nWPAbY0yGseA3xpgMY8FvjDEZxoLfGGMyjAW/McZkGAt+Y4zJMP8/aAqR4i/kKJ4AAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x25224d73b00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stable and comparable result is obtained"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 9792/10000 [============================>.] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.9942\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Deep CNN - 3\n",
    "- One of the ways to reduce the number of parameters in a model is proposed in Min et al 2013 (https://arxiv.org/abs/1312.4400)\n",
    "- Namely \"Network In Network (NIN)\" or \"1 X 1 convolutions\"\n",
    "    - Method is simple - adding convolutional layers after other convolutional layers\n",
    "    - Has the effect of reducing the dimensionality (depth) of image space, effectively reducing the number of parameters\n",
    "    \n",
    "<img src=\"https://raw.githubusercontent.com/iamaaditya/iamaaditya.github.io/master/images/conv_arithmetic/full_padding_no_strides_transposed_small.gif\" style=\"width: 200px\"/>\n",
    "\n",
    "<br>\n",
    "- GoogleNet architecture (Inception) utilizes NIN structure\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/iamaaditya/iamaaditya.github.io/master/images/inception_1x1.png\" style=\"width: 800px\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def deep_cnn_advanced_nin():\n",
    "    model = Sequential()\n",
    "    \n",
    "    model.add(Conv2D(input_shape = (X_train.shape[1], X_train.shape[2], X_train.shape[3]), filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same', kernel_initializer='he_normal'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same', kernel_initializer='he_normal'))\n",
    "    model.add(Conv2D(filters = 25, kernel_size = (1,1), strides = (1,1), padding = 'valid', kernel_initializer='he_normal'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size = (2,2)))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same', kernel_initializer='he_normal'))\n",
    "    model.add(Conv2D(filters = 25, kernel_size = (1,1), strides = (1,1), padding = 'valid', kernel_initializer='he_normal'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same', kernel_initializer='he_normal'))\n",
    "    model.add(Conv2D(filters = 25, kernel_size = (1,1), strides = (1,1), padding = 'valid', kernel_initializer='he_normal'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size = (2,2)))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same', kernel_initializer='he_normal'))\n",
    "    model.add(Conv2D(filters = 25, kernel_size = (1,1), strides = (1,1), padding = 'valid', kernel_initializer='he_normal'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (3,3), strides = (1,1), padding = 'same', kernel_initializer='he_normal'))\n",
    "    model.add(Conv2D(filters = 25, kernel_size = (1,1), strides = (1,1), padding = 'valid', kernel_initializer='he_normal'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size = (2,2)))\n",
    "    \n",
    "    # prior layer should be flattend to be connected to dense layers\n",
    "    model.add(Flatten())\n",
    "    # dense layer with 50 neurons\n",
    "    model.add(Dense(50, activation = 'relu', kernel_initializer='he_normal'))\n",
    "    model.add(Dropout(0.5))\n",
    "    # final layer with 10 neurons to classify the instances\n",
    "    model.add(Dense(10, activation = 'softmax', kernel_initializer='he_normal'))\n",
    "    \n",
    "    adam = optimizers.Adam(lr = 0.001)\n",
    "    model.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['accuracy'])\n",
    "    \n",
    "    return model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = deep_cnn_advanced_nin()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_45 (Conv2D)           (None, 28, 28, 50)        500       \n",
      "_________________________________________________________________\n",
      "batch_normalization_13 (Batc (None, 28, 28, 50)        200       \n",
      "_________________________________________________________________\n",
      "activation_4151 (Activation) (None, 28, 28, 50)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_46 (Conv2D)           (None, 28, 28, 50)        22550     \n",
      "_________________________________________________________________\n",
      "conv2d_47 (Conv2D)           (None, 28, 28, 25)        1275      \n",
      "_________________________________________________________________\n",
      "batch_normalization_14 (Batc (None, 28, 28, 25)        100       \n",
      "_________________________________________________________________\n",
      "activation_4152 (Activation) (None, 28, 28, 25)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_24 (MaxPooling (None, 14, 14, 25)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_48 (Conv2D)           (None, 14, 14, 50)        11300     \n",
      "_________________________________________________________________\n",
      "conv2d_49 (Conv2D)           (None, 14, 14, 25)        1275      \n",
      "_________________________________________________________________\n",
      "batch_normalization_15 (Batc (None, 14, 14, 25)        100       \n",
      "_________________________________________________________________\n",
      "activation_4153 (Activation) (None, 14, 14, 25)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_50 (Conv2D)           (None, 14, 14, 50)        11300     \n",
      "_________________________________________________________________\n",
      "conv2d_51 (Conv2D)           (None, 14, 14, 25)        1275      \n",
      "_________________________________________________________________\n",
      "batch_normalization_16 (Batc (None, 14, 14, 25)        100       \n",
      "_________________________________________________________________\n",
      "activation_4154 (Activation) (None, 14, 14, 25)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_25 (MaxPooling (None, 7, 7, 25)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_52 (Conv2D)           (None, 7, 7, 50)          11300     \n",
      "_________________________________________________________________\n",
      "conv2d_53 (Conv2D)           (None, 7, 7, 25)          1275      \n",
      "_________________________________________________________________\n",
      "batch_normalization_17 (Batc (None, 7, 7, 25)          100       \n",
      "_________________________________________________________________\n",
      "activation_4155 (Activation) (None, 7, 7, 25)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_54 (Conv2D)           (None, 7, 7, 50)          11300     \n",
      "_________________________________________________________________\n",
      "conv2d_55 (Conv2D)           (None, 7, 7, 25)          1275      \n",
      "_________________________________________________________________\n",
      "batch_normalization_18 (Batc (None, 7, 7, 25)          100       \n",
      "_________________________________________________________________\n",
      "activation_4156 (Activation) (None, 7, 7, 25)          0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_26 (MaxPooling (None, 3, 3, 25)          0         \n",
      "_________________________________________________________________\n",
      "flatten_13 (Flatten)         (None, 225)               0         \n",
      "_________________________________________________________________\n",
      "dense_25 (Dense)             (None, 50)                11300     \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 50)                0         \n",
      "_________________________________________________________________\n",
      "dense_26 (Dense)             (None, 10)                510       \n",
      "=================================================================\n",
      "Total params: 87,135\n",
      "Trainable params: 86,785\n",
      "Non-trainable params: 350\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()    # note that the number of parameters are greatly reduced"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 47min 37s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "history = model.fit(X_train, y_train, batch_size = 50, validation_split = 0.2, epochs = 100, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcFNW99/HPb7p79n2BgRlgBkRgRAREBBH3KLhLXFBx\njSHuxniTqHmu5t4kr+vzXJNrTFxC3BPRqAG34JaruERlE0R2kHUGmJ3Zl17O88fpmelZeqaBwYGa\n3/v14sV0VXX3OdXd3zp16lSVGGNQSinVf0T1dQGUUkp9tzT4lVKqn9HgV0qpfkaDXyml+hkNfqWU\n6mc0+JVSqp/R4FdKqX5Gg18ppfoZDX6llOpn3H1dgK5kZmaavLy8vi6GUkodMVasWFFmjMmKZNnD\nMvjz8vJYvnx5XxdDKaWOGCKyI9JltatHKaX6GQ1+pZTqZ3oMfhF5RkRKRGRNmPkiIo+KyBYRWS0i\nE0PmzRCRjcF59/ZmwZVSSh2YSPr4nwP+CLwQZv5MYGTw34nAE8CJIuICHgO+BxQCy0TkTWPMugMp\nqNfrpbCwkMbGxgN5uuogNjaW3NxcPB5PXxdFKfUd6zH4jTGfiEheN4tcBLxg7IX9vxSRVBEZBOQB\nW4wxWwFE5OXgsgcU/IWFhSQlJZGXl4eIHMhLqCBjDOXl5RQWFpKfn9/XxVFKfcd6o48/B9gV8rgw\nOC3c9C6JyFwRWS4iy0tLSzvNb2xsJCMjQ0O/F4gIGRkZuvekVD912BzcNcbMM8ZMMsZMysrqeiiq\nhn7v0XWpVP/VG8FfBAwJeZwbnBZuulJKOUJdk49Xlu9i3e7qg34trz/At6W1vVCqnvXGCVxvArcH\n+/BPBKqMMXtEpBQYKSL52MCfDVzVC+/XJ/bt28f8+fO59dZb9+t55557LvPnzyc1NTXsMg888ACn\nnHIKZ5111sEWU6nvlDGGgAFXVO/tQe6rb2b+0p2MHZzC9JGZrXunxhjW76nhsy2lLN1WyapdlUwa\nls5vLz+OhJj2UdbQ7GfhyiKe+3wbDV4/984Yw7nHZre+Vm2Tj/V7qmn0+mn0BnBHCcfmppCZGNP6\nGv6AYW91Iw3NPhq9Abz+AINS4hiQFIMBXluxi9++v4mSmiZEYPYJQ/jJ90aRlRTTqSw7KuowBgYk\nxZAWH01Uh/W1dncVP311NeV1TXz0b6cRH31oz62Vnm62LiIvAacBmUAx8CDgATDGPCl2Tf4RmAHU\nAzcYY5YHn3su8AjgAp4xxvwmkkJNmjTJdDxzd/369YwZMybiivW27du3c/7557NmTftRrT6fD7f7\nsDwBukd9vU77o2afDZmOP/y+5PXbMu1v919dk4/b5n/Fql37+MG0fK6blkdybPtRYj5/gMLKBnZU\n1LOzop5dFfUU7Wugqt5LVYOXZl+AGWOzuXrKUAYkxfLRhhJ+/vfVlNQ0AXDckFRuOXUEe6oaeGV5\nIev32JZ1fmYCYwYl8d7aYkYNTOKZ608gOyWWPVUNvPDFDl5aupN99V6OGZxMwMD6PdVMH5nJReNz\n+Oe6Yj7aWEKTL9CpTnkZ8YwcmERhZQPfltbS3MUysZ4okmI9lNY0MXFoKnd/72gWbyzl+c+3E+tx\nMTo7Cb8xBAKG4uom9la3P5bmcQkjshKZPjKTk0dmsWJ7BY8v/pbU+Gh+ffExzBg7aL8+hxYissIY\nMymiZXsK/r5wOAb/7NmzeeONNxg1ahQej4fY2FjS0tLYsGEDmzZt4uKLL2bXrl00NjZy1113MXfu\nXKDt8hO1tbXMnDmTk08+mc8//5ycnBzeeOMN4uLiuP766zn//PO59NJLycvL47rrruOtt97C6/Xy\n6quvMnr0aEpLS7nqqqvYvXs3U6dO5YMPPmDFihVkZmYecJ36ep06WSBgEKFda/X1VUX8+u31ZCRG\n88sLj+GkEW2fXaPXT02jj6RYNzHuqHYh3NDs58MNJby9ejdF+xo4f9wgvj8xl4zEGNYUVfHysp18\nU1TNjdPyuPC4wYgIxhj+/lURT3+2jcEpsUwYmsrYnBRqm3zsqmhgV2U928vq2FFez+6qBgYmxXLS\nURmcNCKT6gYvX2wtZ+m2CjyuKI4flsqkYemcdFQGBYOSERFKa5q48bllrNtTzaRhaSzZVkFyrJtz\njsmmvtlPaW0TxdWNFFU24Au0ZUy0O4qc1DjS4j0kx3lo8gb4Yms50a4oxg9JZen2CkYNTOKh7x/L\nhr01PPbRFgorGwAYl5vCZZOGcE7BQAYkxwLw0cYSbn/xK5LjPEwclsa7a/ZijOHsgmxuPDmfE/LS\n8AcMLy7ZycPvb6Sm0UdWUgznjs3mtFEDguvbRYPXz8qdlSzfUcnW0lqGptsNQH5mAokxbmI9LqIE\ndu9rYHt5PXurGjn32EHt9iK2ltbyhw+3UFLTSJQIrighIyGGvIx4hmUm4BKhpKaR4uomVhfuY/n2\nSpr9dsMya2IOD5xfQGp89AF/5xwf/P/x1tpe6VMLVTA4mQcvOCbs/NAW/+LFiznvvPNYs2ZN63DI\niooK0tPTaWho4IQTTuDjjz8mIyOjXfAfddRRLF++nPHjx3P55Zdz4YUXMmfOnE7Bf88993DHHXfw\n+OOP89VXX/HUU09x++23k5OTw3333ce7777LzJkzKS0t1eDHtiol+EPbHxV1zfxzXTHbyusYnpnQ\n+kNPjnW3C15jDLVNPnaU17O9vI7SmiamDM9gdHZSp1ZyS+D+x5trSYp1c8aYAUwdnsn8pTv415Zy\njstNobyumcLKBs47dhDjh6Ty6ZYylmwtb22BelxCnMdFrMdFjCeKsppmGrx+MhNjyEmN5evCKjwu\nYWh6PN+W1hHjjmJwahzbyuo4+ahMfjA9nycWf8vSbRWMzk4K9h3XtStnWryHvMwE8jISGJIWx7dl\ndXy+pYzKei8AQ9PjOTE/HX/AsGxHBbsqbPgOy4hnxjHZvLt2L8XVjTx+9UTOGD2QNUVV/OHDzSzd\nVkFaQjSZiTFkJQVDLyOBYen2/wFJMZ32draW1vLc59v5YF0xF43P4e7vjSTG7QLs3sjijaXkpMZR\nMDi5y89x3e5qbnxuGXVNPq44YQjXnZTHkPT4TsuV1zaxq7KBY3NSerVr6kDVN/tYsq2CpBg3k/LS\nD/r19if4j8w+isPA5MmT242Bf/TRR1m4cCEAu3btYvPmzWRkZLR7Tn5+PuPHjwfg+OOPZ/v27V2+\n9qxZs1qXWbBgAQCfffZZ6+vPmDGDtLS0Xq3PkWhPVQPP/Ws785fupMkXID8jgeFZCYzLTWX6yEwK\nBiW3CxljDN+W1rJ4YykfrCtm2fYKAgaiBEIapcS4o8hMjCExxk1Vg5fK+uYuuwVGZCVw3rGDmDgs\njVHZScR5XPxi4Rr+8c0eTshLIz0hmgVfFfHXL3eSFOvmVxeP5arJQ/H6A/zp4608vngL//hmDyMH\nJHL1icPIy4yntslHbaOPuiYfTb4ATb6AbUmPzebE/AxcUcKm4hpeXrqLdXuquGbKMC6ZkEtirJsX\nl+zgv9/dyA3PLiM13sNDs47l8klDiIoSquq9rNtTTWq8hyHp8STGdP7pBwKGTSU1JMV6yEmNazev\nuLqRDzeU8M6avTz92TaSYt3M/+EUJg6138OxOSn86ZqIMqeT4VmJ/OdFY/nPi8Z2mudxRfG9goHd\nPr9gcDL/e8+piNBt33hGYgwZiTFh53/X4qPdnD5qQJ+89xEZ/N21zL8rCQkJrX8vXryYf/7zn3zx\nxRfEx8dz2mmndTlGPiam7UvncrloaGjo8rVblnO5XPh8vl4ued8pr22ipKapy+Cpa/KxYW8N6/dU\ns62srrW1mJsWT3Wjl71VjeytbmRfvZeqei8lNY18urkMA8wYm01OahxbS2tZv6ead9bs5f++CxkJ\n0YwYkIgruDewrayOon12nY8amMTtpx/F2cdkMyo7iV0V9WwuqWVHeR1ltc2U1TZR2+gjJc5DWkI0\n6QnRDEuPJy8zgZQ4T2vXyx8+2kLLTrMrShDgZzNG8aNTRuCKEpp8fr7eVUV+ZkLrQT9XlIu7zhrJ\nnClD8foN2Smx+7Uejx6YxAMXFHSafu3UPGaMzeYfq/dw4XGD24VcSryHqSMyOj0nVFSUMDq761b1\nwORYrpw8lCsnD6WqwUu0K4q4aNd+lftQ6nhw94D5fVC1E9KH987rHaaOyODvC0lJSdTU1HQ5r6qq\nirS0NOLj49mwYQNffvllr7//tGnTeOWVV/j5z3/O+++/T2VlZa++/sqdlfz0tdWMH5LKT88ZxcDk\nrsPom8IqvtpZyc7gwbqS6kYq671U1jUT47H9tBOGpjE4NdYe2KpqZEd5Hev2VFNc3dT6OgOTYxiU\nEkd1g5ey2iaqG9s2cDHuqC5b2GAPrCXHekiJ83DdSXncMC2P3LT2u/Ul1Y18tqWMTzeXsXtfA75A\ngGY/jM1J5rbTj+LUUVmdWrTDsxIZnpUY8fqaM2UYc6YMo6rey4a91WwsrmFneT0Xjh/MuNy2EVwx\nbheT87vejT8Urc8BSbHcMO3Qno2dEneIL/NhDKx/E4ZNg4QD78rcb8Xr4I1bYfdKuOl/IffA9mCO\nBBr8EcrIyGDatGmMHTuWuLg4Bg5s2/2cMWMGTz75JGPGjGHUqFFMmTKl19//wQcf5Morr+Qvf/kL\nU6dOJTs7m6SkpF557b8t28m/v76W1HgPb67azaJv9nDLqSO44LjBZCRGkxDt5qONJfzpk60s3VYB\n2AAemh7PwORY8jMTSEuIprrBx8pdlfxzfUnrayfFuMlJi2PaiEwmZng5pn4pnyeezdayeoqrG8lN\ni2vtDz56YBIFg5MZnBJLTZOPHWX1FFbWkxLnITslluyU2IiGuQ1IjmXWxFxmTcztufJlW2DrRzDq\nXEgJe2J5WCnxHk4cnsGJw7tvTasIGQOL/g2WPQVp+XDt65CW1zZv5xcQ5YHBE8AV5rsQ8Nv/oyLc\nI6mvsO/38f+D2GTwJMDyZ8IHf/m3sOCHMPV2GDtrv6p3uDgiD+72R00lW3H5G3BnDueL5au45ZZb\nWLVqVYeFasFbDwlZ0OHgpC9g8If+M4YtmzYyf6Md73zyUZn84coJVDd6+e+3VzN68xO48fOvwFhW\nMprmgDA1qYwbjm5k/KijSDnmTMTVdctvX30zZbXNDEyOISl0eN/82bDpHbjxfRh6YtcVLf8W3r0P\nzvolDOzcnbFfakth15fQXA/NtZA9Doac0Da/vgLmnQr7dgICeSfbH/KocyEp++De+2A014O3ARIO\nYGOy7RP4+mUbjKNmQkoEG79AAKI6nMvp98K3H0FMEgyd0u77tN8CfqgugsSB4O5mL8cYWPRTWPZn\nGDcbNr0L7li4ZoGd9979sO1ju2xMsv288k+F4adB1iioK7OBvewpGHiM3WiEs+Nz+OoFKFwG5Vvs\ntLGXwsz/Cx/+2q7DezZAXIfzbxoq4anvQflmiHLDVX+DoyI4/8aYzuvQ12zL2lABU26F+IM7wOv4\nUT1HlKZa8MTaL8mBaqhk84pPuPzmnxMIBIiOS+DxJ/7ECZMn2y9UUw3UFttwA0xaPpWBOCrqvHj9\nAXz+AC2fsosAWbKPGhPP1p27+dFbe/jh9OH89JxRuF1R0FgNL18F2z8lIG6ijA+/uBEMUcbfVqb4\nTDjmEpg8F7KO7rkOmz+AFy+1f4+/Gi5+vOt19fT3oGSdDekffgihG5eti+1ueOUOqNoFqUPtDz//\nlM4/mqZaeHyKXa6FRMGFf4AJc2wYvXgpbP8Mvv8UlGyAb161P2iAnEm2flNu6dxyrN5jy7J1MZRt\nhNPug6PP6brehStgyRP2cxpYAAOOgbxpNlBD1VfY7o2N79jX9TXa+uWeACPPhnFXdB++5d/CBw/A\nhrfBE28bAGDX4+S59vnuaFuObR/DknlQsRXqSux7p+XB8FMhbzoUr4FV8+13CiD7WBtMY7/ffXCH\nCvhtXTa+YwO8vgwQSB5s+8+POtNuYDOPthuZ8i028Jc/Y1vSZ/8aSjfAX2ZBYxX4GmzYn3YfJGbZ\nDdzWxVC53b5f4kAbyv5mSBlqP/efbuncVVSxza6n9W9CXDoMO8m27Ied3NYo2L3KNghm/jecOLft\nuX4v/PX7dqNxxV/gw9/YdXjdm/Y1Gqvs9zN7XPvvY8M+eOEi+/uccA2MvwpKN8I/7rHfHwRiU+C0\ne+GEm9p/5/eDBv/BMgZMIPJdxXDqSqGq0IZ+8mD7RdvPlpO3uRFX2UYajYcdZgD57nJiA/XgicME\n/OD3Ihh8uGmKzSS2uZymgIstgUHEeVzEeVy4XVG4XYJbILF+J26v3UCsK6ph5KBkPLl2pBG1pfDi\n96F4LVz0OIw53+5ab/vU1mFgAQwosCGz5jX7o45JgtuXQVw3o4x8zfDEVPv3kBNhzQL4t432y97C\nGHjtRlj3Opx4M3z5OJz5AEy/x87/4jHb4gOIz4DkHPuja64FBE66Hb73q7b1++598OUTcPkLtvUX\n5Ya3fwzffgjf+0+7gfv0Ybjg93D89W1lKFkHGxfBhn/YH/HJP4GzHmwr59d/g9dvtt+P+AyIToCa\nYrj6FdvybLH9X/DJf9tupNgUG1otG6HkHLj4CRu0AOvfhrfuhPpyG/ajzrPfl6IVtkVaXWSnXfTH\nrluFK1+Et+6yoTz9JzakqwptPVa/CsXfQHIuHH8dbH7fvmbiQLtRSciyn13JersRbK4BcdkN2cRr\nbfh/+YQN4cRsOOkOmHSDrXd33r3PfoaxKXbDNXSKbZFXbrcblr3f2OUSs229A3YYaWvot3yO+3bC\ngh/BoOPg1J91rn/lDrsh2/aprcfkH9rvxLzT4JI/wXGz25ZduxAWzLXfhZN/AlNvg+jOwz4B+3xf\nE9zyuS2LMfD23bDiWfvZjb/Kfu7PnG0DP+MoKPoKjN9ueK5+FQaMtntuf5ll1/ngCVC41L5/wAep\nw+Dch+1e2Xv32+9K5tEw9+Pw5eqGBv/B2rcT6ishdUjnL5qvyX7QjVW2VZZ5dNetoMYqG0zRSfbL\n4K2H6EQbAC08sXZ+yG62MYZGr596r5+GJh/pDTuIwUt5fD7+KA9lNU1ku6pJkXoa/FE04wFPHOX+\nOJr9hkypZrCUU5s0nITE5PbjzKsK7cYoOQeMYf3a1Yx591JwRdtWYsBvv5CXvwBHn93zetqz2v5A\nxl9lQymcfz0KH/w7XP2aDcs/nw7n/da2blp8/kd4/xdw5oM2vP52DWx6D27+DHb8y4b2mAvhosds\nPyzYFljRCljxHHz9kg3+aXfaaU+dBZNutO/T+tk1w8IfwVo7RJaJ19o9gK4YY8P0q+fh0mdsa3fT\n+/DylTB0Ksz4L9t6b9wHz51nA+2a121r7cNf2Q1MQlYwKG+0G8jGKhsA7/zctnCn3m6nrfyLbSVe\n8AgMnti+cWCMDd4PHrDdT99/un032dI/2z7x4afBJfMgaWDnemz5p90I7VpiQ+nkH9u9Lk+HA/h+\nH+xdDUmDIHlQ+9f49kP47H9g+6f2Mxx7qQ27tGE20BJDhiXuXmU/4wnX2PXfVQu2qsh2++380n4f\nBx5j9ywG9MJvPBCA346yXUGXPdtWhydOsn/PWdC+fl1Z8Zz9/G98H3KOt+t4xbOdGwIVW+HFy+xG\nZ/hpkDXahriv0f6OlvzJNpAufdp+h0o3waoX7fdhyq1tAW+M3SgXrYDT7z+gamvwH4y6cjucK8pj\nWyEJWbb11VhtQzPYnYI71n64yTntv/QAzXX2h+2OgYyRtouhvhyqd9uNQDtREJOEP2EA5c1uyuua\n8QbP5hsUVUkW+/AmD8OTaDdANY1edlXYkSrJsR4GpcYS43ZhjKHJF8Dr85G4byMSkwTpIaM7WvY+\nErJa+33Xr1vLmKqPoXav7Vf2N9sw3J/RDB88AP/6PVz3NuRP7zy/ejf8cbLt3rjqb/YL/uR0G243\nf2qX2fwBzL/C9klf8Vc7r6YYHj/Rbij37bT9qLPn2+6KjgIB+PuNtkV3yTz4/FG7vm9b0n6vAuzG\n7Z8P2oO6lz3XOfxC+Zrh+Qtgz9dwzm/gvV9A5ki4/h9tGx+wZX12pl2//ia7Zzf9J3bD5onr/LrN\ndfD+v8PypwGBk++2XRhd1a1F0Qp49QbYt8NueCZeC7Ulti5Hz+y5LsbYjVNK7gF3JQCwcwl8+lvb\n1eILDkeOTrIt3GFT7fp96kz7ud++rPP6/668fhtseAt+utUeBG7pvjn/f+yGuCdNtfDb0bZLyltv\nQ/nku+GMBzofD+mocofdGJRttI/PfdjuiRxiGvwHyltvt8jRCZA+Amp228CUqGDXj8f2Gcal2VAv\nXhcM9xFtrxHw2d1mibJ7A6E/MmPAGKoavRRVNhBrGkiNqieZOjCGTSaX2JgY0hKiSZBmPJWbkfgM\nu/sfwue3J/aEHbtcVWT7bgcU2NZ8XRlUF9oQTR/e2qLslXXaXG+7ccRld4s9sbaeu5bAiudtGJsA\n3PpF23pqaaXOXWx/YC9eatdVx0D9+mXbQs+bboOlqxBt4W2Ev1wCOz+3j6940XZVHazaEph3ul1/\n6cPhxvc6b+jBhv7rt8DQk2wXQmzX4+Hb2f6ZrVPO8ZGVpbHKrtOvnm87IFlwMcz6c/cbjUPBGLtu\nyrfYbqrq3XDly1C2yX62338ajr30uy1TqHVvwCvXwvWLbKNj0c9sK/7fNnU+YBvO23fbYw4SZfdc\nItlgtGiohLd+bD/baXceUBX2lwZ/V0zAHgRt2Gd3rxI6XPM/4LcHXEzAjhBoCez6CvuDi0uF2NT2\nu+H7dtkj8tnj2qbXl9sWasZIiGk/LtwYw96qRkprm4iPdpMS56bJG8D4Gsnx7cREJ+LKGAEYW5aA\n3/YT7u+BYV+z7auOT7cbosYqG/ppee2OW/TaxvTbD23o5k233S8l66GpynZtHXspnPBDyA45K7Nh\nn21NDTnB9oum5NofaMdRLMbYcMyZ2HOfMtjP6oUL7QZv1ryDr1eLPattV8nZv2obWtiXjLEHGEvW\nwfE3hB/W+F2pLbEHL8u/tb+b3BPgmoUHNxLoYDVWw/8bDlNvhdN/Ybt+hp/e1vUTidJNsOAm+/xw\nB+8PI/sT/IfNjVgOqdoS2LvG9sc1VNgWcKdliu1uelpe+1Z6fLrtMolL6/xFjkm0G4qWERRgQy3K\nQ2KGHQ5YWFjExbO+T3F1I1tKaimtbSIjIYbhWQlcdsEM9m5dx5AB6UQlD8bVXGM3HDXFwVEdQ3jk\n0T9SX9/2+ueeey779u3rvr7uaLuhqi+3P4DkHNtaPdiD1eGMOMO2hvZ+Y9fRsZfaA2D3bLQHT7M7\nnIoflwrHXGy7CxKybN94V0MXRWz3USShD/azmvuJPajXmwaNs6M4DofQB7te8qbZ7oO+Dn2we0DX\nvW0bKQGfbR339Y1+YpNt19Om9+2xooZKe1xjf2QdDT/65IgI/f11GHxrvgN1ZTbMk4fZg3GNXVzg\nzddo++1jIj97k+jgkLymGhtOAb/9OziErKiynn2SyH8++jTF1Y3EelwMSY8nrasr8CVk2ZZ5dZFt\n0cWlQWwKjzzyCHPmzCE+3h4EWrRoUWRlS8y25UnKjjw4D8b5/2P/Rerku+3xkrN/0/OBtv3RU/+r\nOjQSMmw3WF2ZHRRxOBh5jh0w8OnD9oD1iNP7ukSHjf7xKwn4bLdDbIrt8w74bEs9lN9r+/DDuPfe\ne3nsscdaH//yl7/k1//1EGdecSsTp53Jsccey+uvvQQYdjfFEDBQUe+lunQ3s8+ZRsGgZIYku7nl\nxmsZM2YMl1xySbtr9dxy661MOmc2x5w2iwd/+yQk5/Loo4+ye/duTj/9dE4/3X5p8/LyKCuzeyy/\n+93vGDt2LGPHjuWRRx4B7FVEx4wZww9vvYNjTrmQs8+/JOw1gfpU1ih7IDdtWF+XRPUWT9zhE/rQ\n1lLf87U9j+FQ7fEegY7MFv8797aNA+6RsS1LVzS4YuxIHV+jPS1bQrZ7yYPhjP8T9lWuuOIKfvzj\nH3PbbbcB8Morr/DGW4v40ZxZZEY3s64hnQvOms65n75JtT8aERiTnUShN44oEdyuKB594gni4+NZ\nv349q1evZuLEia2v/5vf/Ib09HT8jbWcec65rF67jjvvvJPf/e53fPTRR50uv7xixQqeffZZlixZ\ngjGGE088kVNPPZW0tDQ2b97MSy+9xJ///Gcuv/xy/v73vzNnzpyIV69SjpBxlL3sQ+U2O+RYtToy\ng/+ASIf/Qw9qB0/Y6maY24QJEygpKWH37t0Ul5SQkJRCdVQCv/rlf7Bq6Wf4JJo9e4vZW93MqDHJ\nCNgzYUN88skn3HmnPcI/btw4xo0b1zrvlVdeYd68efh8Pvbs2cO6devaze/os88+45JLLmm9Suis\nWbP49NNPufDCCyO+/LNSjiZiz7zevdLuYapWR2bwz3wo8mW9jVC63p4lF59uhx+WbbQtgZZhXX6v\nPZswTFePMQYR4bLLLuOVV19l07ZCzjjvYj59ZyH1dbUsf2c+nrhE8o8/k4Ardr9vYbdt2zYefvhh\nli1bRlpaGtdff32Xl3WOVKSXf1bK8U78UV+X4LDk/D7+lhOmWoZEtrTqW04RD/27Q4vfDr9sYM3u\najburWH6ORfwwl/ns+jNhVx39RWIt4FB2dlEJ6Sy+ONP2VG4p9sDqaeccgrz588HYM2aNaxevRqA\n6upqEhISSElJobi4mHfeeaf1OeEuBz19+nRef/116uvrqaurY+HChUyf3sUJVEop1cGR2eLfH4Hg\ndd5bDuy0bAD8IcHvDy7TIfiLq+2NQ5JjPYjA0BGjqa2tZeiQXApG5DHg6qu54IILOPbUC5k0diSj\nR45of9ygg1tuuYUbbriBMWPGMGbMGI4/3p64c9xxxzFhwgRGjx7NkCFDmDZtWutz5s6dy4wZMxg8\neDAfffRR6/SJEydy/fXXM3nyZABuuukmJkyYoN06SqkeOf8ErpYTqgYUtF1TZ+8aiE2y3T9gh6BV\n7Wq3THF1I8XVjaQnRJOTGtd9901L91H6iMjO2DxMHLFXPFVKdaL33A3VscUP9qSXllZ+6DLBFn9Z\nbRPF1Y3wfPIhAAAXSElEQVSkxUcQ+mDPBB54jB05pJRShznn9/EH/IDYa8m0iPJ06Orx2vkSRXWD\nl937GkiJ85CbFkHot9DQV0odISIKfhGZISIbRWSLiNzbxfw0EVkoIqtFZKmIjA2Zd7eIrBWRNSLy\nkojs352lQ3TbLWWMvbZGbUn76QGfbe2HBrjL0/7grt8LLg8NzX52VtQT53GRmxa/36NzjiSHYxef\nUuq70WPwi4gLeAyYCRQAV4pIx3vi3Q+sMsaMA64Ffh98bg5wJzDJGDMWcAGzOQCxsbGUl5eHDywR\ne60dX4dhkAFf54ucRXmCZ+8GXyvgJRDlYXt5Ha4oIS8zAVeUs0O/vLyc2NgD3gYrpY5gkfTxTwa2\nGGO2AojIy8BFwLqQZQqAhwCMMRtEJE9EWu4I4QbiRMQLxAO7D6Sgubm5FBYWUlpaGn6hmjKIqoSE\nurZpLXsA5SHLNdXai7VVrsVrBFftHhqNh31Uk5kYw5Z9zu8Bi42NJTc3gvuxKqUcJ5LgzwFCblxK\nIdDxTtlfA7OAT0VkMjAMyDXGrBCRh4GdQAPwvjHm/QMpqMfjIT8/v/uFXnwQava03eAD4PEb7ZUp\nZ7/YNm3DInj9Sl4Y+xy/XO5mU8y1fDJgDsMu+y+OGrAfF2lTSqkjUG81bR8CUkVkFXAHsBLwi0ga\ndu8gHxgMJIhIlxeNEZG5IrJcRJZ326rvTlI21OxtP62+ovP9YIO3p/vkq2+4emwcbglwxgnjNPSV\nUv1CJMFfBIReci83OK2VMabaGHODMWY8to8/C9gKnAVsM8aUGmO8wALgpK7exBgzzxgzyRgzKSsr\nq6tFepY0yN4xq2XEjjG2Sye+/bXem+LsHZSOjq/j/ulpbc9VSql+IJLgXwaMFJF8EYnGHpx9M3QB\nEUkNzgO4CfjEGFON7eKZIiLxYofInAms773id5A8CDD2pipg723qb+50w/THl9cQMMJloz3ENQb3\nLpKyD1mxlFLqcNJj8BtjfMDtwHvY0H7FGLNWRG4WkZuDi40B1ojIRuzon7uCz10CvAZ8BXwTfL9e\nvCdeBy2t9pbunvrgEd24tuDfVFzD4x9vp86dSn50tT0mABr8Sql+I6Izd40xi4BFHaY9GfL3F8DR\nYZ77IPDgQZQxci3hXR0cONRQYf8PafH/5h/rSYxxE5eRY/cMWvYOEgeilFL9gbPGLXZq8bcEv+3j\n31vVyCebS7lmah7u5EF2uZo99raH3VyLXymlnMRZwR+faU/Waum+aai0/we7el5fVYQxMGtCjh3Z\nU7PX/tNuHqVUP+Ks4I+KsjcZ79jHH5+OMYYFXxUycWgqeZkJwRFAJVBVpCN6lFL9irOCH4Jj+YN9\n/C1dPbGprN1dzabiWmZNDJ6tmjjQ3m6xbKO2+JVS/YpDgz/Y4m+ogNgUcLlZuLKIaFcU548b1LYc\n2OGe2uJXSvUjzgv+5MFtffz19uQtnz/AG6uKOGP0AFLjg6cbhIa9tviVUv2I84I/KRsaq+xdserL\nIS6dTzeXUVbbzCUTc9qWCx2+qS1+pVQ/4sDgbxnSuSd4uYZ0FqwsIjXew+mjBrQtFxr8OoZfKdWP\nODD4g902NXuhvpJAXBqLN5ZwTkE20e6Q6rqj267hoy1+pVQ/4sDgH2z/D7b4ywOJ1DT6OOmojM7L\nJmaDRNkTuJRSqp9w3s3WW1r8lduhuZZtdTEATB3RRfAnDbTHAVzOWw1KKRWO8xIvNgXccVBibxC2\ndp+bkQMSGZDUxW0Gj7sSKnd8xwVUSqm+5bzgF7Gt/mIb/F+XRTF1UhetfYBxl3+HBVNKqcOD8/r4\nwY7lL98MQLE/gZO66uZRSql+ypnBn5QNAR8A+0jixHwNfqWUauHQ4G8bnpk1IJu0hOhuFlZKqf7F\nocHfdgmGMcPz+q4cSil1GHJo8NsWf72J4cSjB/dxYZRS6vDi6OCvJIkT8tJ7WFgppfoXhwa/7epp\n9KSQFKu3VFRKqVAODX7b4vfHpPVxQZRS6vDjzOCPjqeaBBo9KX1dEqWUOuw478zdoMe4nMGZExjX\n1wVRSqnDTEQtfhGZISIbRWSLiNzbxfw0EVkoIqtFZKmIjA2Zlyoir4nIBhFZLyJTe7MC4TzvP4ei\n9MnfxVsppdQRpcfgFxEX8BgwEygArhSRgg6L3Q+sMsaMA64Ffh8y7/fAu8aY0cBxwPreKHhPfH6D\nO0q+i7dSSqkjSiQt/snAFmPMVmNMM/AycFGHZQqADwGMMRuAPBEZKCIpwCnA08F5zcaYfb1W+jCM\nMfgCBo/LmYcwlFLqYESSjDnArpDHhcFpob4GZgGIyGRgGJAL5AOlwLMislJEnhKRhIMudQ+8fgOA\nx6UtfqWU6qi3msQPAakisgq4A1gJ+LEHjycCTxhjJgB1QKdjBAAiMldElovI8tLS0oMqjNcfANAW\nv1JKdSGSZCwChoQ8zg1Oa2WMqTbG3GCMGY/t488CtmL3DgqNMUuCi76G3RB0YoyZZ4yZZIyZlJV1\ncLdC9AVb/G4NfqWU6iSSZFwGjBSRfBGJBmYDb4YuEBy503IJzJuAT4Ibg73ALhEZFZx3JrCul8oe\nVnOwxR+tXT1KKdVJj+P4jTE+EbkdeA9wAc8YY9aKyM3B+U8CY4DnRcQAa4EfhLzEHcCLwQ3DVuCG\nXq5DJ76ADX5t8SulVGcRncBljFkELOow7cmQv78Ajg7z3FXApIMo437z+loO7mrwK6VUR45MRm+g\n5eCudvUopVRHzgx+HdWjlFJhOTIZW0f16Jm7SinViSODv2VUj8ftyOoppdRBcWQyen3B4I9yZPWU\nUuqgODIZfQG9ZINSSoXjyOBv6erRcfxKKdWZI5Ox5eButAa/Ukp14shk9La2+LWrRymlOnJ08Os4\nfqWU6syRyajX41dKqfAcGfw+bfErpVRYjkxG7epRSqnwHJmM2tWjlFLhOTT4tcWvlFLhODIZW87c\n1eGcSinVmSODv1mv1aOUUmE5Mhl9gQCuKCFKL8uslFKdODL4vX6jB3aVUioMhwZ/QLt5lFIqDEem\no9cf0JuwKKVUGI5MR5/f6G0XlVIqDEcGf7M/oGP4lVIqDEemo08P7iqlVFgRBb+IzBCRjSKyRUTu\n7WJ+mogsFJHVIrJURMZ2mO8SkZUi8nZvFbw7Xm3xK6VUWD2mo4i4gMeAmUABcKWIFHRY7H5glTFm\nHHAt8PsO8+8C1h98cSPj9Ru97aJSSoURSTpOBrYYY7YaY5qBl4GLOixTAHwIYIzZAOSJyEAAEckF\nzgOe6rVS98DrDxCtXT1KKdWlSII/B9gV8rgwOC3U18AsABGZDAwDcoPzHgF+BgS6exMRmSsiy0Vk\neWlpaQTFCs8XCGiLXymlwuitdHwISBWRVcAdwErALyLnAyXGmBU9vYAxZp4xZpIxZlJWVtZBFcbr\n04O7SikVjjuCZYqAISGPc4PTWhljqoEbAEREgG3AVuAK4EIROReIBZJF5K/GmDm9UPawvIEAiZ5I\nqqaUUv1PJC3+ZcBIEckXkWhgNvBm6AIikhqcB3AT8IkxptoYc58xJtcYkxd83oeHOvRBR/UopVR3\nemwWG2N8InI78B7gAp4xxqwVkZuD858ExgDPi4gB1gI/OIRl7pGeuauUUuFF1B9ijFkELOow7cmQ\nv78Aju7hNRYDi/e7hAegWa/Vo5RSYTkyHX1+g0db/Eop1SVHBr/28SulVHiOTEc9c1cppcJzZDrq\nmbtKKRWeI4Pf59czd5VSKhxHpqO9564jq6aUUgfNcelojMEbCOglG5RSKgzHBb8/YDAGbfErpVQY\njktHX8AA4NYWv1JKdclxwd/st1d/jtYWv1JKdclx6ejzB1v8euauUkp1yXHB7w22+PVaPUop1TXH\npWNr8Ec5rmpKKdUrHJeO3mBXj8etXT1KKdUVxwW/L9jid2uLXymluuS4dGwZ1aPj+JVSqmuOS8eW\nUT165q5SSnXNccHv1Ra/Ukp1y3Hp2HJwV8/cVUqprjkw+PXMXaWU6o7j0tEXCI7q0eBXSqkuOS4d\nm316cFcppbrjuOBvafHrwV2llOpaROkoIjNEZKOIbBGRe7uYnyYiC0VktYgsFZGxwelDROQjEVkn\nImtF5K7erkBHOqpHKaW612M6iogLeAyYCRQAV4pIQYfF7gdWGWPGAdcCvw9O9wH3GGMKgCnAbV08\nt1d59eqcSinVrUiaxZOBLcaYrcaYZuBl4KIOyxQAHwIYYzYAeSIy0BizxxjzVXB6DbAeyOm10neh\ndVSPXp1TKaW6FEk65gC7Qh4X0jm8vwZmAYjIZGAYkBu6gIjkAROAJQdW1Mjo9fiVUqp7vdUsfghI\nFZFVwB3ASsDfMlNEEoG/Az82xlR39QIiMldElovI8tLS0gMuiF6PXymluueOYJkiYEjI49zgtFbB\nML8BQEQE2AZsDT72YEP/RWPMgnBvYoyZB8wDmDRpkom8Cu219PHrCVxKKdW1SNJxGTBSRPJFJBqY\nDbwZuoCIpAbnAdwEfGKMqQ5uBJ4G1htjftebBQ/H23pZZu3qUUqprvTY4jfG+ETkduA9wAU8Y4xZ\nKyI3B+c/CYwBnhcRA6wFfhB8+jTgGuCbYDcQwP3GmEW9XI9WPn8AEXBp8CulVJci6eohGNSLOkx7\nMuTvL4Cju3jeZ8B3msDNfoMnKgq7s6GUUqojx3WE+/wBvVyDUkp1w3HB7/UH9AJtSinVDcclpDdg\n9HINSinVDcclpNenXT1KKdUdxwW/T1v8SinVLcclZLM/oLddVEqpbjgu+H3+gJ61q5RS3XBcQnr9\nRlv8SinVDQcGf0D7+JVSqhuOS0ivP4AnynHVUkqpXuO4hPT6DR63dvUopVQ4jgt+nz+AW1v8SikV\nluMSstmv4/iVUqo7jktIvUibUkp1z3HBr6N6lFKqe45LSB3Hr5RS3XNg8OuZu0op1R3HJaQvoC1+\npZTqjuOC316W2XHVUkqpXuO4hPQGNPiVUqo7jktIr9/ocE6llOqGo4I/EDD4A0bP3FVKqW44KiG9\ngQAA0W5HVUsppXpVRAkpIjNEZKOIbBGRe7uYnyYiC0VktYgsFZGxkT63N/n8BgB3lHb1KKVUOD0G\nv4i4gMeAmUABcKWIFHRY7H5glTFmHHAt8Pv9eG6v8fpti18P7iqlVHiRJORkYIsxZqsxphl4Gbio\nwzIFwIcAxpgNQJ6IDIzwub3GG2zx68FdpZQKL5LgzwF2hTwuDE4L9TUwC0BEJgPDgNwIn9trtMWv\nlFI9662EfAhIFZFVwB3ASsC/Py8gInNFZLmILC8tLT2gQrT28WvwK6VUWO4IlikChoQ8zg1Oa2WM\nqQZuABARAbYBW4G4np4b8hrzgHkAkyZNMpEVv73m1ha/dvUopVQ4kTSNlwEjRSRfRKKB2cCboQuI\nSGpwHsBNwCfBjUGPz+1NvoB29SilVE96bPEbY3wicjvwHuACnjHGrBWRm4PznwTGAM+LiAHWAj/o\n7rmHpirg9bUc3NXgV0qpcCLp6sEYswhY1GHakyF/fwEcHelzD5WWE7j06pxKKRWeo5rGXl/wzF1t\n8SulVFiOSkhfQM/cVUqpnjgq+FtH9ei1epRSKixHJWTLOH6PXp1TKaXCclRCtp6569auHqWUCseR\nwa/X41dKqfAclZAtF2nTUT1KKRWeoxLS59dx/Eop1RNHBb9enVMppXrmqITU6/ErpVTPHBb82uJX\nSqmeOCohW87c1eBXSqnwHJWQzT69Hr9SSvXEUcHvCwRwRwn2XjBKKaW64qjg9/qNDuVUSqkeOCz4\nA9q/r5RSPXBUSmrwK6VUzxyVkj6/0QO7SinVA0cFf7M/oBdoU0qpHjgqJX1+Q7TehEUppbrlqJT0\n+gN620WllOqBw4Lf6MFdpZTqgaNS0o7q0Ra/Ukp1x1HB7wvocE6llOpJRCkpIjNEZKOIbBGRe7uY\nnyIib4nI1yKyVkRuCJl3d3DaGhF5SURie7MCobw+PXNXKaV60mPwi4gLeAyYCRQAV4pIQYfFbgPW\nGWOOA04Dfisi0SKSA9wJTDLGjAVcwOxeLH87Xm3xK6VUjyJJycnAFmPMVmNMM/AycFGHZQyQJPbq\naIlABeALznMDcSLiBuKB3b1S8i7ombtKKdWzSFIyB9gV8rgwOC3UH4Ex2FD/BrjLGBMwxhQBDwM7\ngT1AlTHm/a7eRETmishyEVleWlq6n9Ww9MxdpZTqWW81j88BVgGDgfHAH0UkWUTSsHsH+cF5CSIy\np6sXMMbMM8ZMMsZMysrKOqBCNPsDuLXFr5RS3YokJYuAISGPc4PTQt0ALDDWFmAbMBo4C9hmjCk1\nxniBBcBJB1/srvn8hmgNfqWU6lYkKbkMGCki+SISjT04+2aHZXYCZwKIyEBgFLA1OH2KiMQH+//P\nBNb3VuE70jN3lVKqZ+6eFjDG+ETkduA97KicZ4wxa0Xk5uD8J4FfAc+JyDeAAD83xpQBZSLyGvAV\n9mDvSmDeoalK8MxdvVaPUkp1q8fgBzDGLAIWdZj2ZMjfu4Gzwzz3QeDBgyhjxLz+AB5t8SulVLcc\n1Tz26XBOpZTqkaNS8uxjsikYnNzXxVBKqcNaRF09R4r/uWJ8XxdBKaUOe45q8SullOqZBr9SSvUz\nGvxKKdXPaPArpVQ/o8GvlFL9jAa/Ukr1Mxr8SinVz2jwK6VUPyPGmL4uQyciUgrsOMCnZwJlvVic\nI0F/rDP0z3r3xzpD/6z3/tZ5mDEmopuZHJbBfzBEZLkxZlJfl+O71B/rDP2z3v2xztA/630o66xd\nPUop1c9o8CulVD/jxOA/ZDd6OYz1xzpD/6x3f6wz9M96H7I6O66PXymlVPec2OJXSinVDccEv4jM\nEJGNIrJFRO7t6/IcKiIyREQ+EpF1IrJWRO4KTk8XkQ9EZHPw/7S+LmtvExGXiKwUkbeDj/tDnVNF\n5DUR2SAi60VkqtPrLSJ3B7/ba0TkJRGJdWKdReQZESkRkTUh08LWU0TuC+bbRhE552De2xHBLyIu\n4DFgJlAAXCkiBX1bqkPGB9xjjCkApgC3Bet6L/C/xpiRwP8GHzvNXcD6kMf9oc6/B941xowGjsPW\n37H1FpEc4E5gkjFmLOACZuPMOj8HzOgwrct6Bn/js4Fjgs95PJh7B8QRwQ9MBrYYY7YaY5qBl4GL\n+rhMh4QxZo8x5qvg3zXYIMjB1vf54GLPAxf3TQkPDRHJBc4DngqZ7PQ6pwCnAE8DGGOajTH7cHi9\nsXcGjBMRNxAP7MaBdTbGfAJUdJgcrp4XAS8bY5qMMduALdjcOyBOCf4cYFfI48LgNEcTkTxgArAE\nGGiM2ROctRcY2EfFOlQeAX4GBEKmOb3O+UAp8Gywi+spEUnAwfU2xhQBDwM7gT1AlTHmfRxc5w7C\n1bNXM84pwd/viEgi8Hfgx8aY6tB5xg7VcsxwLRE5HygxxqwIt4zT6hzkBiYCTxhjJgB1dOjicFq9\ng33aF2E3eoOBBBGZE7qM0+oczqGsp1OCvwgYEvI4NzjNkUTEgw39F40xC4KTi0VkUHD+IKCkr8p3\nCEwDLhSR7dhuvDNE5K84u85gW3WFxpglwcevYTcETq73WcA2Y0ypMcYLLABOwtl1DhWunr2acU4J\n/mXASBHJF5Fo7EGQN/u4TIeEiAi2z3e9MeZ3IbPeBK4L/n0d8MZ3XbZDxRhznzEm1xiTh/1sPzTG\nzMHBdQYwxuwFdonIqOCkM4F1OLveO4EpIhIf/K6fiT2O5eQ6hwpXzzeB2SISIyL5wEhg6QG/izHG\nEf+Ac4FNwLfAL/q6PIewnidjd/9WA6uC/84FMrCjADYD/wTS+7qsh6j+pwFvB/92fJ2B8cDy4Of9\nOpDm9HoD/wFsANYAfwFinFhn4CXscQwvdu/uB93VE/hFMN82AjMP5r31zF2llOpnnNLVo5RSKkIa\n/Eop1c9o8CulVD+jwa+UUv2MBr9SSvUzGvxKKdXPaPArpVQ/o8GvlFL9zP8HQVz9HnVj0vwAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x252255d5978>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fairly stable result even with smaller number of parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 2s     \n"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.9914\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: ', results[1])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
